https://giov.dev/ GIOVANNI CARMANTINI 2019-03-16T16:51:44+00:00 My personal website, with the occasional blog post, contact informations and various projects. Giovanni Carmantini Jekyll https://giov.dev/2018/05/a-window-on-numpy-s-views.html A window on Numpy's views 2018-05-20T20:37:59+00:00 Giovanni A breathtaking panorama for numeric Python enthusiasts and professionals. <p>If you use Numpy for your everyday number crunching and you’ve never heard about views, you are not as effective as you could be. With this post, I want to help you take your Numpy skills to the next level.</p> <p>Your mental model of what arrays are in Numpy is probably quite different from the underlying reality. <!--more--> To get you to the paradigm shift, we need a problem that can bring us down the rabbit hole. This is the <em>windowing problem</em>:</p> <blockquote> <p>I have an input time series with <script type="math/tex">k</script> observations. I want to rearrange it in <script type="math/tex">w</script> windows of some size <script type="math/tex">s</script> and apply a function <script type="math/tex">f</script> to each window, to get an output array of <script type="math/tex">w</script> values. How can I do this fast and with a small memory footprint using Numpy?</p> </blockquote> <p>Or – more visually – how to use Numpy to do something like the following, while keeping fast and small?</p> <div class="videofigure"> <video class="player" loop="" poster="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/window_apply.jpg" preload="metadata" width="100%" height="100%" style="width=100%;height=100%;"> <source src="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/window_apply.mp4" type="video/mp4" /> </video> </div> <p>Two naive solutions to windowing end up being either too slow or too memory-hungry.</p> <p>Specifically, using a pure Python <code class="highlighter-rouge">for</code> loop to iterate window-by-window over the original array is going to be slow; Creating a new windowed array from the old one requires up to <script type="math/tex">w + 1</script> times the amount of memory used by the old array. Both of these approaches could work for a one-off script on a small dataset, but they become unfeasible as soon as the data size increases, or our data transformation needs to be called repeatedly in some data processing pipeline.</p> <p>Enter views.</p> <h2 id="data-views-in-numpy">Data views in numpy</h2> <p>A numpy array is a pointer to a contiguous chunk of memory, bundled with information on how to interpret that memory. It turns out that you can modify this information to reinterpret the same contiguous chunk of memory, and <em>see</em> it in a different way. When you do that, you are creating a <em>view</em>. <sup id="fnref:numpy_paper"><a href="#fn:numpy_paper" class="footnote">1</a></sup> The original array is then akin to a “canonical” interpretation of the underlying memory, and it <em>owns</em> that memory. A view reinterprets the canon, but it doesn’t own its memory – it shares it with the original array.</p> <p>The interpretation information is contained in key <code class="highlighter-rouge">numpy.ndarray</code> attributes:</p> <ul> <li><em>data</em>: a pointer to the chunk of memory</li> <li><em>dtype</em>: data type. Is each byte in the memory chunk representing a boolean, an 8-bit integer, or perhaps is part of the 8 bytes forming a 64-bit floating point number?</li> <li><em>itemsize</em>: number of bytes needed to represent a single element, e.g. 8 for a 64-bit float. This is included for clarity, as it is actually completely defined by dtype.</li> <li><em>shape</em>: the size of the array in each of the dimensions. For example, a 2D array with <script type="math/tex">j</script> rows and <script type="math/tex">k</script> columns would have <code class="highlighter-rouge">shape=(j, k)</code>.</li> <li><em>strides</em>: for each dimension, the number of bytes to jump ahead to get to the next element in that dimension. More about this later.</li> </ul> <p>Here’s an example of how this information comes together to form an interpretation of some memory chunk:</p> <figure class="image"> <img class="center with-border" src="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/numpy_data_view.svg" id="numpy-data-view" /> <figcaption class="center">Numpy view</figcaption> </figure> <p>By modifying the interpretation information bundled with every numpy array, we can keep the same underlying memory contents, yet</p> <ul> <li>see the memory as containing one type of data or another (e.g. integers, floats, booleans, or arbitrary python objects);</li> <li>decide how to move through this memory, ignoring some parts of it and only accessing some other;</li> </ul> <p>Numpy’s view approach is extremely powerful. To illustrate why, say I have a 1GB array <code class="highlighter-rouge">big_arr</code>, and I want to create another array <code class="highlighter-rouge">smaller_arr</code> from it, containing all elements in the second half of <code class="highlighter-rouge">big_arr</code>. By creating a view of the original <code class="highlighter-rouge">big_arr</code>, numpy will just reuse the same memory from <code class="highlighter-rouge">big_arr</code> instead of copying half of its contents to some newly allocated 500MB memory chunk.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">bytes_in_gigabyte</span> <span class="o">=</span> <span class="mf">1e9</span> <span class="o">&gt;&gt;&gt;</span> <span class="c"># This increases RAM usage by 1GB (check with a RAM monitor)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">big_arr</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="n">bytes_in_gigabyte</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">int8</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="c"># No big increase in RAM for smaller_arr</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">smaller_arr</span> <span class="o">=</span> <span class="n">big_arr</span><span class="p">[</span><span class="n">big_arr</span><span class="o">.</span><span class="n">size</span><span class="o">/</span><span class="mi">2</span><span class="p">:]</span> </code></pre> </div> <p>This is interesting and all, but – you may be wondering – what does this have to do with windowing? Let’s talk about <code class="highlighter-rouge">strides</code>.</p> <h2 id="moving-through-memory">Moving through memory</h2> <p>As I wrote before, a numpy array contains information about how to move through the memory chunk which the array points to.</p> <p>An array’s <code class="highlighter-rouge">strides</code> tuple <script type="math/tex">(i_0, i_1, \ldots, i_N)</script> is the answer to the question “If I am moving along dimension <script type="math/tex">j</script>, how many bytes of memory do I have to skip to get to the next element?”. For example, for the array of 1-byte integers in <a href="#numpy-data-view">the figure</a> with 2 rows and 4 columns, <code class="highlighter-rouge">strides</code> would be equal to <code class="highlighter-rouge">(4, 1)</code>, as if I am moving along dimension 0 (i.e. along rows), I need to skip 4 bytes of data to get to the next element from the current one, and to skip 1 byte to get to the next element if I am moving along dimension 1 (i.e. columns). An animation should help clarifying the concept:</p> <div class="videofigure"> <video class="player" loop="" poster="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/striding_normal.jpg" preload="metadata" width="100%" height="100%" style="width=100%;height=100%;"> <source src="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/striding_normal.mp4" type="video/mp4" /> </video> </div> <p>We can modify <code class="highlighter-rouge">strides</code> to get a memory-efficient windowed view of an array, and be able to apply fast numerical transformations to it. This is because we can rely on Numpy’s fast vectorization to apply functions to our windowed array rather than having to use slow pure Python.</p> <h3 id="windowing-with-strides">Windowing with strides</h3> <p>Let’s take the chunk of memory in <a href="#numpy-data-view">the previous figure</a>, interpreted as a sequence of 1-byte integers:</p> <p><code class="highlighter-rouge">[1, 3, 3, 7, 8, 0, 0, 8]</code></p> <p>Say we want to window this such that each window contains 2 elements, with the windowing advancing by one element at a time; this would result in the following array:</p> <p><code class="highlighter-rouge">[[1, 3], [3, 3], [3, 7], [7, 8], [8, 0], [0, 0], [0, 8]]</code></p> <p>If we were to create the windowed array from scratch, we would end up with an array roughly 2 times as big as the original one. As I wrote before, this is not practical when one has a big array to begin with, and the problem only grows worse as the windowing size increases. Luckily, we can manipulate the <code class="highlighter-rouge">strides</code> and <code class="highlighter-rouge">shape</code> information to move intelligently through the original chunk of memory, simulating windowing but without the memory cost. Remember, <code class="highlighter-rouge">strides</code> contains, for each dimension, the number of bytes to jump ahead to get to the next element in that dimension.</p> <p>For the chunk of memory in the example, we can iterate over it as if it were windowed, by setting its shape to be <code class="highlighter-rouge">(7, 2)</code>, and specifying that the first dimension always advances by one element at a time (one byte in the case of a int8 element). In this way, every time we are finished looking at a row (a window), we advance to the next one by moving a byte forward from the beginning of the current row, and see the next couple of elements as the next row.</p> <div class="videofigure"> <video class="player" loop="" poster="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/striding_windowed.jpg" preload="metadata" width="100%" height="100%" style="width=100%;height=100%;"> <source src="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/striding_windowed.mp4" type="video/mp4" /> </video> </div> <p>Interestingly, Numpy won’t actually just allow us to change strides and shape without putting up a fight, so we need to use the <code class="highlighter-rouge">as_strided</code> function under <code class="highlighter-rouge">numpy.lib.slide_tricks</code>, which circumvents numpy’s safety checks to create an array on the original data with our desired shape and strides.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">arr</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">8</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">int8</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">np</span><span class="o">.</span><span class="n">lib</span><span class="o">.</span><span class="n">stride_tricks</span><span class="o">.</span><span class="n">as_strided</span><span class="p">(</span><span class="n">arr</span><span class="p">,</span> <span class="n">strides</span><span class="o">=</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">7</span><span class="p">,</span><span class="mi">2</span><span class="p">))</span> <span class="n">array</span><span class="p">([[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">7</span><span class="p">],</span> <span class="p">[</span><span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">],</span> <span class="p">[</span><span class="mi">8</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">8</span><span class="p">]],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">int8</span><span class="p">)</span> </code></pre> </div> <p>A word of warning: bypassing numpy’s checks on shape and strides, we can do some serious damage. For example, what happens when the new shape makes the view “spill over” the original memory chunk?</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">np</span><span class="o">.</span><span class="n">lib</span><span class="o">.</span><span class="n">stride_tricks</span><span class="o">.</span><span class="n">as_strided</span><span class="p">(</span><span class="n">arr</span><span class="p">,</span> <span class="n">strides</span><span class="o">=</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">))</span> <span class="n">array</span><span class="p">([[</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">7</span><span class="p">],</span> <span class="p">[</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">8</span><span class="p">],</span> <span class="p">[</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="p">[</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">8</span><span class="p">],</span> <span class="p">[</span> <span class="mi">8</span><span class="p">,</span> <span class="o">-</span><span class="mi">16</span><span class="p">],</span> <span class="p">[</span> <span class="o">-</span><span class="mi">16</span><span class="p">,</span> <span class="o">-</span><span class="mi">108</span><span class="p">],</span> <span class="p">[</span><span class="o">-</span><span class="mi">108</span><span class="p">,</span> <span class="mi">30</span><span class="p">]],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">int8</span><span class="p">)</span> </code></pre> </div> <p>That’s strange. What are those weird values starting from the 8th row? We never defined those. Those value are memory we never allocated for our original array, and that we were not supposed to access. Who knows who is using it and what it actually contains.</p> <p>To avoid getting garbage in our arrays and messing with memory that is not ours to mess with, we must be sure to compute shapes and strides properly while windowing.</p> <h2 id="efficient-time-series-windowing-in-numpy">Efficient time series windowing in Numpy</h2> <p>The following function contains all the needed logic to make sure we can window efficiently and safely:</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">division</span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> <span class="k">def</span> <span class="nf">sliding_window</span><span class="p">(</span><span class="n">arr</span><span class="p">,</span> <span class="n">window_size</span><span class="p">,</span> <span class="n">step</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span> <span class="s">"""Assuming a time series with time advancing along dimension 0, window the time series with given size and step. :param arr : input array. :type arr: numpy.ndarray :param window_size: size of sliding window. :type window_size: int :param step: step size of sliding window. If 0, step size is set to obtain non-overlapping contiguous windows (that is, step=window_size). Defaults to 0. :type step: int :return: array :rtype: numpy.ndarray """</span> <span class="n">n_obs</span> <span class="o">=</span> <span class="n">arr</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c"># validate arguments</span> <span class="k">if</span> <span class="n">window_size</span> <span class="o">&gt;</span> <span class="n">n_obs</span><span class="p">:</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span> <span class="s">"Window size must be less than or equal "</span> <span class="s">"the size of array in first dimension."</span> <span class="p">)</span> <span class="k">if</span> <span class="n">step</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">:</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">"Step must be positive."</span><span class="p">)</span> <span class="n">n_windows</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="nb">int</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">floor</span><span class="p">(</span> <span class="p">(</span><span class="n">n_obs</span> <span class="o">-</span> <span class="n">window_size</span><span class="p">)</span> <span class="o">/</span> <span class="n">step</span><span class="p">))</span> <span class="n">obs_stride</span> <span class="o">=</span> <span class="n">arr</span><span class="o">.</span><span class="n">strides</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">windowed_row_stride</span> <span class="o">=</span> <span class="n">obs_stride</span> <span class="o">*</span> <span class="n">step</span> <span class="n">new_shape</span> <span class="o">=</span> <span class="p">(</span><span class="n">n_windows</span><span class="p">,</span> <span class="n">window_size</span><span class="p">)</span> <span class="o">+</span> <span class="n">arr</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="n">new_strides</span> <span class="o">=</span> <span class="p">(</span><span class="n">windowed_row_stride</span><span class="p">,</span> <span class="p">)</span> <span class="o">+</span> <span class="n">arr</span><span class="o">.</span><span class="n">strides</span> <span class="n">strided</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">lib</span><span class="o">.</span><span class="n">stride_tricks</span><span class="o">.</span><span class="n">as_strided</span><span class="p">(</span> <span class="n">arr</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="n">new_shape</span><span class="p">,</span> <span class="n">strides</span><span class="o">=</span><span class="n">new_strides</span><span class="p">,</span> <span class="p">)</span> <span class="k">return</span> <span class="n">strided</span> </code></pre> </div> <p>Let’s go through this bit by bit.</p> <p>The first few lines are not particularly interesting, we just validate the passed arguments to make sure they make sense.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code> <span class="n">n_obs</span> <span class="o">=</span> <span class="n">arr</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c"># validate arguments</span> <span class="k">if</span> <span class="n">window_size</span> <span class="o">&gt;</span> <span class="n">n_obs</span><span class="p">:</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span> <span class="s">"Window size must be less than or equal "</span> <span class="s">"the size of array in first dimension."</span> <span class="p">)</span> <span class="k">if</span> <span class="n">step</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">:</span> <span class="k">raise</span> <span class="nb">ValueError</span><span class="p">(</span><span class="s">"Step must be positive."</span><span class="p">)</span> </code></pre> </div> <p>Subsequently, we count the number of windows that the windowed array will end up containing.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code> <span class="n">n_windows</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="nb">int</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">floor</span><span class="p">(</span> <span class="p">(</span><span class="n">n_obs</span> <span class="o">-</span> <span class="n">window_size</span><span class="p">)</span> <span class="o">/</span> <span class="n">step</span><span class="p">))</span> </code></pre> </div> <p>To understand the equation, think about this way. We place a window at the beginning of the array. How many times we can slide this window forwards by <code class="highlighter-rouge">step</code> before the window goes over the array bounds? That’s equal to the number of times we can fit a step in <code class="highlighter-rouge">n_obs - window_size</code>, i.e. <code class="highlighter-rouge">floor( (n_obs - window_size)/step )</code>. For a more visual intuition:</p> <div class="videofigure"> <video class="player" loop="" poster="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/window_animation.jpg" preload="metadata" width="100%" height="100%" style="width=100%;height=100%;"> <source src="https://giov.dev/assets/posts/2018-05-20-a-window-on-numpy-s-views/window_animation.mp4" type="video/mp4" /> </video> </div> <p>We are finally ready to create the windowed view. Whereas the old array was organized as a sequence of observations, the view is organized as a sequence of windows, each one containing <code class="highlighter-rouge">window_size</code> observations. We need to update the <code class="highlighter-rouge">shape</code> metadata to reflect this change in dimensionality, as well as <code class="highlighter-rouge">strides</code>, to let Numpy know how many bytes it needs to skip to go from one window to the next.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code> <span class="n">obs_stride</span> <span class="o">=</span> <span class="n">arr</span><span class="o">.</span><span class="n">strides</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">windowed_row_stride</span> <span class="o">=</span> <span class="n">obs_stride</span> <span class="o">*</span> <span class="n">step</span> <span class="n">new_shape</span> <span class="o">=</span> <span class="p">(</span><span class="n">n_windows</span><span class="p">,</span> <span class="n">window_size</span><span class="p">)</span> <span class="o">+</span> <span class="n">arr</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="n">new_strides</span> <span class="o">=</span> <span class="p">(</span><span class="n">windowed_row_stride</span><span class="p">,</span> <span class="p">)</span> <span class="o">+</span> <span class="n">arr</span><span class="o">.</span><span class="n">strides</span> <span class="n">strided</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">lib</span><span class="o">.</span><span class="n">stride_tricks</span><span class="o">.</span><span class="n">as_strided</span><span class="p">(</span> <span class="n">arr</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="n">new_shape</span><span class="p">,</span> <span class="n">strides</span><span class="o">=</span><span class="n">new_strides</span><span class="p">,</span> <span class="n">writeable</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="p">)</span> <span class="k">return</span> <span class="n">strided</span> </code></pre> </div> <p>And we are done. Let’s see it in action.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">big_arr</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="mf">1e9</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">int8</span><span class="p">)</span> </code></pre> </div> <p>Again, <code class="highlighter-rouge">big_arr</code> is an array of size ~1GB.</p> <div class="language-python highlighter-rouge"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="n">sl_arr</span> <span class="o">=</span> <span class="n">sliding_window</span><span class="p">(</span><span class="n">big_arr</span><span class="p">,</span> <span class="n">window_size</span><span class="o">=</span><span class="mi">1000</span><span class="p">,</span> <span class="n">step</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">out</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">empty</span><span class="p">(</span><span class="n">sl_arr</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">int8</span><span class="p">)</span> <span class="o">&gt;&gt;&gt;</span> <span class="n">np</span><span class="o">.</span><span class="nb">min</span><span class="p">(</span><span class="n">sl_arr</span><span class="p">,</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">out</span><span class="p">)</span> </code></pre> </div> <p>If we were to actually create the windowed array from scratch, we would need ~1TB worth of memory to accommodate it. Thanks to the power of views, the new array only needs enough memory to contain the interpretation information (11 orders of magnitude less), and is created instantly.</p> <h2 id="conclusion">Conclusion</h2> <p>Numpy is great at hiding unnecessary complexity (such as <code class="highlighter-rouge">strides</code> or memory bounds) from day-to-day work. Yet, from time to time, it is good to open the clock and see what makes it tick. Windowing is the problem that had me open the clock. In this post, I used the windowing problem to abuse Numpy’s views a little and show you their inner workings, the same way I learned about them. Hope you enjoyed it.</p> <p>In case you came here through a specific search for efficient windowing in Numpy, I can point you to some other useful resources.</p> <p>Throughout this post, windowing was only applied to the first dimension of an array. This is because I assumed a time series as input, where iteration over subsequent observations is normally done on the first array dimension. That is, <code class="highlighter-rouge">array[i]</code> contains the value of some measurement(s) at time <script type="math/tex">t_i</script>. If time series are not where you spend most of your number crunching, then your windowing needs might be different. You might want to have a look at a more general windowing implementations, such as <a href="https://gist.github.com/meowklaski/4bda7c86c6168f3557657d5fb0b5395a">this one</a> or <a href="https://gist.github.com/nils-werner/9d321441006b112a4b116a8387c2280c">this one</a>.</p> <p>If you <em>are</em> looking for time series windowing in Python, but you are not comfortable tinkering with stride tricks, consider using <a href="https://pandas.pydata.org">Pandas</a> in your projects, and have a look at its <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.rolling.html">rolling interface</a>.</p> <p>Finally, if you feel the need for even more advanced windowing capabilities than covered here or the links provided, you may find what you are looking for in <a href="https://numba.pydata.org/">Numba</a> or <a href="http://cython.org/">Cython</a>.</p> <p>If you enjoyed this post, I’d be grateful if you’d help it spread by sharing it with your fellow data crunchers and numeric Python enthusiasts. I’d also love to read what you think, so go on and make a blogger happy – drop me a comment!</p> <div class="footnotes"> <ol> <li id="fn:numpy_paper"> <p>See <a href="https://arxiv.org/abs/1102.1523">this paper</a> for an introduction <a href="#fnref:numpy_paper" class="reversefootnote">&#8617;</a></p> </li> </ol> </div> 2018-05-20T20:37:59+00:00 https://giov.dev/2016/06/deriving-the-power-of-wald-test-for-a-single-parameter.html Deriving the power of Wald test for a single parameter 2016-06-22T15:56:00+00:00 Giovanni A small post proving a key result relating to the Wald test. <p>While studying from Larry Wasserman’s “All of Statistics”, I’ve found that the exposition of the Wald test was a little confusing to me, so that I struggled a bit in trying to derive a key result. Given that I didn’t find much on the internet to help me, and that I finally figured it out after a while, I thought of writing a small post for other confused students.</p> <!--more--> <p>Given a scalar parameter <script type="math/tex">\theta</script> of the distribution underlying the data, the Wald test uses its estimate <script type="math/tex">\hat{\theta}</script> to compute a statistic <script type="math/tex">{W}</script>, which is then used to pit a null hypothesis <script type="math/tex">H_0</script> against an alternative hypothesis <script type="math/tex">H_1</script>. The distribution of the estimate <script type="math/tex">\hat{\theta}</script> is assumed to be asymptotically normal, centered on the true value of <script type="math/tex">\theta</script>. The null hypothesis is <script type="math/tex">H_0: \theta = \theta_0</script>, which states that the true value of the parameter <script type="math/tex">\theta</script> is some scalar <script type="math/tex">\theta_0</script> (so that, if we assume that <script type="math/tex">H_0</script> is true and given that <script type="math/tex">\hat{\theta}</script> is asymptotically normal, <script type="math/tex">\hat{\theta}</script> would be centered on <script type="math/tex">\theta_0</script>). The alternative hypothesis is instead <script type="math/tex">H_1: \theta \ne \theta_0</script>, which states the opposite.</p> <p>Let <script type="math/tex">\widehat{\text{se}}</script> be the estimated standard error of <script type="math/tex">\hat{\theta}</script>. If the null hypothesis is true, and thus the asymptotic distribution of <script type="math/tex">\hat{\theta}</script> is a normal centered on <script type="math/tex">\theta_0</script>, then</p> <script type="math/tex; mode=display">\begin{equation} \frac{\hat{\theta} - \theta_0}{\widehat{\text{se}}} \rightsquigarrow N(0,1), \end{equation}</script> <p>where <script type="math/tex">{N}(0,1)</script> is the normal distribution with mean 0 and unit standard deviation.</p> <p>The Wald statistic is defined as</p> <script type="math/tex; mode=display">\begin{equation} W = \frac{\hat{\theta} - \theta_0}{\widehat{\text{se}}} \end{equation}</script> <p>and the size <script type="math/tex">\alpha</script> Wald test states: reject the null hypothesis when <script type="math/tex">\vert W \vert >z_\frac{\alpha}{2}</script>, where <script type="math/tex">z_\frac{\alpha}{2}</script> is equal to <script type="math/tex">\Phi^{-1}(1-\frac{\alpha}{2})</script>, <script type="math/tex">\Phi^{-1}</script> being the inverse of the normal CDF. <script type="math/tex">{W}</script> is asymptotically distributed as <script type="math/tex">{N}(0,1)</script>, so that the probability of rejecting the null hypothesis asymptotically converges to <script type="math/tex">\alpha</script>.</p> <p>Now for the key result I wanted to prove:</p> <p><strong>Theorem</strong>: Let <script type="math/tex">\theta_\ast</script> be the true value of <script type="math/tex">\theta</script>, <script type="math/tex">\theta_\ast \ne \theta_0</script> (i.e. the null hypothesis really is false). The power <script type="math/tex">\beta(\theta_\ast) = P_{\theta_\ast}(\vert W \vert >z_\frac{\alpha}{2})</script> of correctly rejecting the null hypothesis is then approximately equal to</p> <script type="math/tex; mode=display">\begin{equation} 1 - \Phi\left(\frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} + z_\frac{\alpha}{2}\right) + \Phi\left(\frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} - z_\frac{\alpha}{2}\right) \end{equation}</script> <p>In proving this result I was initially confused by the fact that in Wasserman’s book <script type="math/tex">\hat{\theta}</script> is assumed to be asymptotically normal with center in <script type="math/tex">\theta_0</script> (Theorem 10.3). I was then further led astray by <a href="http://math.stackexchange.com/questions/1478667/proof-of-the-power-of-the-wald-test">this question</a> on math.stackexchange, where the user asking the question incorrectly applies the definition of power of a test.</p> <p>So here’s the proof:</p> <p><strong>Proof</strong>: The true value of <script type="math/tex">\theta</script> is <script type="math/tex">\theta_\ast</script>: given that <script type="math/tex">\hat{\theta}</script> is asymptotically normal and centered on <script type="math/tex">\theta_\ast</script>, then <script type="math/tex">\frac{\hat{\theta} - \theta_\ast}{\widehat{\text{se}}} \rightsquigarrow N(0,1)</script>. The power of the Wald test for <script type="math/tex">\theta=\theta_\ast</script> is equal to</p> <script type="math/tex; mode=display">% <![CDATA[ \begin{align} \beta(\theta_\ast) & = P_{\theta_\ast}(\vert W \vert>z_\frac{\alpha}{2}) \\ & = P_{\theta_\ast}\left(\frac{\vert \hat\theta - \theta_0\vert}{\widehat{\text{se}}} > z_\frac{\alpha}{2}\right)\\ & = P_{\theta_\ast}\left(\frac{\hat\theta - \theta_0}{\widehat{\text{se}}} > z_\frac{\alpha}{2} \right) + P_{\theta_\ast}\left(\frac{\hat\theta - \theta_0}{\widehat{\text{se}}} < - z_\frac{\alpha}{2} \right) \\ & = P_{\theta_\ast}\left(\frac{\hat\theta - \theta_0}{\widehat{\text{se}}} + \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} > \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} + z_\frac{\alpha}{2} \right) +\\ & \hphantom{{}={}} P_{\theta_\ast}\left(\frac{\hat\theta - \theta_0}{\widehat{\text{se}}} + \frac{\theta_0 - \theta_*}{\widehat{\text{se}}} < \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} - z_\frac{\alpha}{2} \right)\\ \end{align} %]]></script> <p>let <script type="math/tex">Z = \frac{\hat\theta - \theta_0}{\widehat{\text{se}}} + \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} = \frac{\hat{\theta} - \theta_\ast}{\widehat{\text{se}}}</script>, then <script type="math/tex">Z \rightsquigarrow N(0,1)</script>, so that</p> <script type="math/tex; mode=display">% <![CDATA[ \begin{align} & \hphantom{{}={}}P_{\theta_\ast}\left(Z > \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} + z_\frac{\alpha}{2}\right) + P_{\theta_\ast}\left(Z < \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} - z_\frac{\alpha}{2}\right) \\\\ & = 1 - P_{\theta_\ast}\left(Z < \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} + z_\frac{\alpha}{2}\right) + P_{\theta_\ast}\left(Z < \frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} - z_\frac{\alpha}{2}\right) \\ & = 1 - \Phi\left(\frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} + z_\frac{\alpha}{2}\right) + \Phi\left(\frac{\theta_0 - \theta_\ast}{\widehat{\text{se}}} - z_\frac{\alpha}{2}\right) \end{align} %]]></script> <p>concluding the proof <script type="math/tex">\blacksquare</script></p> 2016-06-22T15:56:00+00:00 https://giov.dev/2016/01/minimum-edit-distance-in-python.html Minimum Edit Distance in Python 2016-01-21T11:50:00+00:00 Giovanni Some notes on the use of dynamic programming to compute the minimum edit distance between two strings in Python. <p>While I’m going through the <a href="https://class.coursera.org/nlp/lecture">NLP course by Jurafsky and Manning</a> on coursera, I coded a small python implementation of the Wagner-Fischer algorithm presented in <a href="https://class.coursera.org/nlp/lecture/6">lecture 6</a>, <a href="https://class.coursera.org/nlp/lecture/7">7</a> and <a href="https://class.coursera.org/nlp/lecture/8">8</a>. And here it is! Please refer to the lectures for a more in-depth explanations of the algorithm. I’ll just go quickly through the basics and then present the code.</p> <!--more--> <h2 id="introduction">Introduction</h2> <blockquote> <p>How similar are these two strings?</p> </blockquote> <p>Many people from different fields often end up asking themselves this question: the computational biologist comparing sequences of bases to see if they contain similar information; the computer scientist implementing speech recognition, trying to make sense of odd recognition results; me fighting with autocorrection for the control of my smartphone.</p> <p>To actually answer this question, you first need to define some concept of distance between strings.</p> <p>A useful definition is that of <strong>edit distance</strong>:</p> <blockquote> <p>The <strong>edit distance</strong> is the number of operations (insertions, deletions or substitutions) needed to transform one string in another.</p> </blockquote> <p>where an insert operation means adding a symbol to the string, deletion means subtracting one, and substitution is a deletion followed by an insertion. Depending on your definition of edit distance, you may just consider insertion and deletion and do without substitution.</p> <p>For example, we may want to calculate the distance between the strings <em>spell</em> and <em>help</em>:</p> <table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> <colgroup> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> </colgroup> <tr> <td class="org-left"> s </td> <td class="org-left"> p </td> <td class="org-left"> e </td> <td class="org-left"> l </td> <td class="org-left"> l </td> </tr> <tr> <td class="org-left"> h </td> <td class="org-left"> e </td> <td class="org-left"> l </td> <td class="org-left"> p </td> <td class="org-left"> &#xa0; </td> </tr> </table> <p>One way to transform <em>help</em> into <em>spell</em> is to align the two <em>el</em> substrings, insert an <em>s</em> at the beginning of <em>help</em>, and perform the remaining substitutions. If we abbreviate insertion, deletion and substitution with i, d and s,</p> <table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> <colgroup> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> </colgroup> <tr> <th scope="col" class="org-left"> s </th> <th scope="col" class="org-left"> p </th> <th scope="col" class="org-left"> e </th> <th scope="col" class="org-left"> l </th> <th scope="col" class="org-left"> l </th> </tr> <tr> <th scope="col" class="org-left"> &#xa0; </th> <th scope="col" class="org-left"> h </th> <th scope="col" class="org-left"> e </th> <th scope="col" class="org-left"> l </th> <th scope="col" class="org-left"> p </th> </tr> <tr> <td class="org-left"> i </td> <td class="org-left"> s </td> <td class="org-left"> &#xa0; </td> <td class="org-left"> &#xa0; </td> <td class="org-left"> s </td> </tr> </table> <p>so that, depending on whether we consider substitution to be a single operation or two, we end up with an edit distance between <em>spell</em> and <em>help</em> of respectively 3 or 5.</p> <h2 id="minimum-edit-distance">Minimum edit distance</h2> <p>Normally we are not interested in <em>any</em> edit distance, but we want the <strong>minimum</strong> edit distance between two strings. How to compute it?</p> <p>There is an infinity of ways in which we can transform one string into another. We can get creative with alignments, inserting whole books’ worth of characters and then deleting the ones we don’t need, hiring a chiliad of monkeys randomly tapping on a keyboard until they manage to get from the first string to the second, and so on.</p> <p>Luckily, if it is the <strong>minimum</strong> edit distance that we want, we don’t need to search this enormous space naively; we can be smart about it.</p> <p>Say we have an initial state (the starting string) and an ending state (the final string). To go from one to the other, we apply a sequence of operations: a path connecting them. It turns out that to find the shortest path between two states, we just need to make sure that we are following the shortest possible path between each of the intermediate states between the two.</p> <p>This problem can be solved elegantly by dynamic programming.</p> <h3 id="dynamic-programming-for-minimum-edit-distance-wagnerfischer-algorithm">Dynamic Programming for Minimum Edit Distance: Wagner–Fischer algorithm</h3> <p>With dynamic programming, we solve a large problem by first solving smaller parts of it, and then building on the knowledge we gathered to solve bigger and bigger parts.</p> <p>In the Wagner-Fischer algorithm, we define a distance matrix <script type="math/tex">D_{i,j} = d(X[1\ldots i], Y[1\ldots j])</script>, the matrix in which index <script type="math/tex">(i, j)</script> corresponds to the minimum edit distance <script type="math/tex">d</script> between the first <script type="math/tex">i</script> symbols in <script type="math/tex">X</script> and the first <script type="math/tex">j</script> symbols in <script type="math/tex">Y</script>. We first compute <script type="math/tex">D_{i,j}</script> for small <script type="math/tex">(i,j)</script>, and then go for larger and larger <script type="math/tex">i</script> and <script type="math/tex">j</script> using the smaller bits that we already computed before.</p> <p>By doing this, we end up with the minimum edit distance between <script type="math/tex">X</script> and <script type="math/tex">Y</script>, that is <script type="math/tex">d(X[1\ldots m], Y[1 \ldots n])</script>, where <script type="math/tex">m = \vert X \vert</script> is the length of the <script type="math/tex">X</script> string, and <script type="math/tex">n = \vert Y \vert</script> is the length of the <script type="math/tex">Y</script> string.</p> <p>I’ll now go through a python implementation of the algorithm. I’ll be using python3, as I wanted unicode support and I didn’t want to deal with the unicode nonsense in python2. To run the code in python2, just take out the unicode arrows.</p> <p>First things first, let’s import some libraries. Numpy just makes things cleaner (not much going on here in terms of numerics), and we use tabulate to produce the final tables for backtracking and alignment.</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2</pre></td><td class="code"><pre><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span> <span class="kn">import</span> <span class="nn">tabulate</span> <span class="kn">as</span> <span class="nn">tb</span> <span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> <p>We jump straight into defining the key function:</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30</pre></td><td class="code"><pre><span class="k">def</span> <span class="nf">wagner_fischer</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">word_2</span><span class="p">):</span> <span class="n">n</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">word_1</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span> <span class="c"># counting empty string </span> <span class="n">m</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">word_2</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span> <span class="c"># counting empty string</span> <span class="c"># initialize D matrix</span> <span class="n">D</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">m</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="nb">int</span><span class="p">)</span> <span class="n">D</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="n">D</span><span class="p">[</span><span class="mi">0</span><span class="p">,:]</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">)</span> <span class="c"># B is the backtrack matrix. At each index, it contains a triple</span> <span class="c"># of booleans, used as flags. if B(i,j) = (1, 1, 0) for example,</span> <span class="c"># the distance computed in D(i,j) came from a deletion or a</span> <span class="c"># substitution. This is used to compute backtracking later.</span> <span class="n">B</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">m</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="p">[(</span><span class="s">"del"</span><span class="p">,</span> <span class="s">'b'</span><span class="p">),</span> <span class="p">(</span><span class="s">"sub"</span><span class="p">,</span> <span class="s">'b'</span><span class="p">),</span> <span class="p">(</span><span class="s">"ins"</span><span class="p">,</span> <span class="s">'b'</span><span class="p">)])</span> <span class="n">B</span><span class="p">[</span><span class="mi">1</span><span class="p">:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="n">B</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">:]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">l_1</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span> <span class="k">for</span> <span class="n">j</span><span class="p">,</span> <span class="n">l_2</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">word_2</span><span class="p">,</span> <span class="n">start</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span> <span class="n">deletion</span> <span class="o">=</span> <span class="n">D</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">j</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span> <span class="n">insertion</span> <span class="o">=</span> <span class="n">D</span><span class="p">[</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span> <span class="n">substitution</span> <span class="o">=</span> <span class="n">D</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="p">(</span><span class="mi">0</span> <span class="k">if</span> <span class="n">l_1</span><span class="o">==</span><span class="n">l_2</span> <span class="k">else</span> <span class="mi">2</span><span class="p">)</span> <span class="n">mo</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="nb">min</span><span class="p">([</span><span class="n">deletion</span><span class="p">,</span> <span class="n">insertion</span><span class="p">,</span> <span class="n">substitution</span><span class="p">])</span> <span class="n">B</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">deletion</span><span class="o">==</span><span class="n">mo</span><span class="p">,</span> <span class="n">substitution</span><span class="o">==</span><span class="n">mo</span><span class="p">,</span> <span class="n">insertion</span><span class="o">==</span><span class="n">mo</span><span class="p">)</span> <span class="n">D</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">mo</span> <span class="k">return</span> <span class="n">D</span><span class="p">,</span> <span class="n">B</span><span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> <p>And here we implement a naive backtrace:</p> <ul> <li>start from index <script type="math/tex">(m,n)</script>,</li> <li>look at where the computed value in <script type="math/tex">(m,n)</script> came from</li> <li>In order of preference, follow a substitution, or a deletion, or an insertion (that is, go to the cell up and to the left if that’s where the value in <script type="math/tex">(m,n)</script> was computed from, or to the cell above, or to the cell to the left)</li> </ul> <figure class="highlight"><pre><code class="language-python" data-lang="python"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2 3 4 5 6 7 8 9 10 11 12 13 14</pre></td><td class="code"><pre><span class="k">def</span> <span class="nf">naive_backtrace</span><span class="p">(</span><span class="n">B_matrix</span><span class="p">):</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="n">B_matrix</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">B_matrix</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">-</span><span class="mi">1</span> <span class="n">backtrace_idxs</span> <span class="o">=</span> <span class="p">[(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">)]</span> <span class="k">while</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">)</span> <span class="o">!=</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">):</span> <span class="k">if</span> <span class="n">B_matrix</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">][</span><span class="mi">1</span><span class="p">]:</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">j</span><span class="o">-</span><span class="mi">1</span> <span class="k">elif</span> <span class="n">B_matrix</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">][</span><span class="mi">0</span><span class="p">]:</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">j</span> <span class="k">elif</span> <span class="n">B_matrix</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">][</span><span class="mi">2</span><span class="p">]:</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="o">-</span><span class="mi">1</span> <span class="n">backtrace_idxs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">))</span> <span class="k">return</span> <span class="n">backtrace_idxs</span><span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> <p>This next function takes a backtrace and computes the alignment between the two words. It goes through the operations and takes note of what has been applied at each step, while constructing the alignment.</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39</pre></td><td class="code"><pre><span class="k">def</span> <span class="nf">align</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">word_2</span><span class="p">,</span> <span class="n">bt</span><span class="p">):</span> <span class="n">aligned_word_1</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">aligned_word_2</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">operations</span> <span class="o">=</span> <span class="p">[]</span> <span class="n">backtrace</span> <span class="o">=</span> <span class="n">bt</span><span class="p">[::</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="c"># make it a forward trace</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">backtrace</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">):</span> <span class="n">i_0</span><span class="p">,</span> <span class="n">j_0</span> <span class="o">=</span> <span class="n">backtrace</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="n">i_1</span><span class="p">,</span> <span class="n">j_1</span> <span class="o">=</span> <span class="n">backtrace</span><span class="p">[</span><span class="n">k</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="n">w_1_letter</span> <span class="o">=</span> <span class="bp">None</span> <span class="n">w_2_letter</span> <span class="o">=</span> <span class="bp">None</span> <span class="n">op</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">if</span> <span class="n">i_1</span> <span class="o">&gt;</span> <span class="n">i_0</span> <span class="ow">and</span> <span class="n">j_1</span> <span class="o">&gt;</span> <span class="n">j_0</span><span class="p">:</span> <span class="c"># either substitution or no-op</span> <span class="k">if</span> <span class="n">word_1</span><span class="p">[</span><span class="n">i_0</span><span class="p">]</span> <span class="o">==</span> <span class="n">word_2</span><span class="p">[</span><span class="n">j_0</span><span class="p">]:</span> <span class="c"># no-op, same symbol</span> <span class="n">w_1_letter</span> <span class="o">=</span> <span class="n">word_1</span><span class="p">[</span><span class="n">i_0</span><span class="p">]</span> <span class="n">w_2_letter</span> <span class="o">=</span> <span class="n">word_2</span><span class="p">[</span><span class="n">j_0</span><span class="p">]</span> <span class="n">op</span> <span class="o">=</span> <span class="s">" "</span> <span class="k">else</span><span class="p">:</span> <span class="c"># cost increased: substitution</span> <span class="n">w_1_letter</span> <span class="o">=</span> <span class="n">word_1</span><span class="p">[</span><span class="n">i_0</span><span class="p">]</span> <span class="n">w_2_letter</span> <span class="o">=</span> <span class="n">word_2</span><span class="p">[</span><span class="n">j_0</span><span class="p">]</span> <span class="n">op</span> <span class="o">=</span> <span class="s">"s"</span> <span class="k">elif</span> <span class="n">i_0</span> <span class="o">==</span> <span class="n">i_1</span><span class="p">:</span> <span class="c"># insertion</span> <span class="n">w_1_letter</span> <span class="o">=</span> <span class="s">" "</span> <span class="n">w_2_letter</span> <span class="o">=</span> <span class="n">word_2</span><span class="p">[</span><span class="n">j_0</span><span class="p">]</span> <span class="n">op</span> <span class="o">=</span> <span class="s">"i"</span> <span class="k">else</span><span class="p">:</span> <span class="c"># j_0 == j_1, deletion</span> <span class="n">w_1_letter</span> <span class="o">=</span> <span class="n">word_1</span><span class="p">[</span><span class="n">i_0</span><span class="p">]</span> <span class="n">w_2_letter</span> <span class="o">=</span> <span class="s">" "</span> <span class="n">op</span> <span class="o">=</span> <span class="s">"d"</span> <span class="n">aligned_word_1</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">w_1_letter</span><span class="p">)</span> <span class="n">aligned_word_2</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">w_2_letter</span><span class="p">)</span> <span class="n">operations</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">op</span><span class="p">)</span> <span class="k">return</span> <span class="n">aligned_word_1</span><span class="p">,</span> <span class="n">aligned_word_2</span><span class="p">,</span> <span class="n">operations</span><span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> <p>Finally, this function formats the results from the Wagner–Fischer algorithm and backtracking to a table that is human-readable. In the table, each cell contains the computed minimum edit distance from the initial state to that state, and where it was computed from (that is, what operations could have produced it). The backtrace is highlighted with asterisks.</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32</pre></td><td class="code"><pre><span class="k">def</span> <span class="nf">make_table</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">word_2</span><span class="p">,</span> <span class="n">D</span><span class="p">,</span> <span class="n">B</span><span class="p">,</span> <span class="n">bt</span><span class="p">):</span> <span class="n">w_1</span> <span class="o">=</span> <span class="n">word_1</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="n">w_2</span> <span class="o">=</span> <span class="n">word_2</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span> <span class="n">w_1</span> <span class="o">=</span> <span class="s">"#"</span> <span class="o">+</span> <span class="n">w_1</span> <span class="n">w_2</span> <span class="o">=</span> <span class="s">"#"</span> <span class="o">+</span> <span class="n">w_2</span> <span class="n">table</span> <span class="o">=</span> <span class="p">[]</span> <span class="c"># table formatting in emacs, you probably don't need this line</span> <span class="n">table</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="s">"&lt;r&gt;"</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">w_2</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">)])</span> <span class="n">table</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="s">""</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">w_2</span><span class="p">))</span> <span class="n">max_n_len</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="nb">max</span><span class="p">(</span><span class="n">D</span><span class="p">)))</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">l_1</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">w_1</span><span class="p">):</span> <span class="n">row</span> <span class="o">=</span> <span class="p">[</span><span class="n">l_1</span><span class="p">]</span> <span class="k">for</span> <span class="n">j</span><span class="p">,</span> <span class="n">l_2</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">w_2</span><span class="p">):</span> <span class="n">v</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">h</span> <span class="o">=</span> <span class="n">B</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">]</span> <span class="n">direction</span> <span class="o">=</span> <span class="p">(</span><span class="s">"⇑"</span> <span class="k">if</span> <span class="n">v</span> <span class="k">else</span> <span class="s">""</span><span class="p">)</span> <span class="o">+</span>\ <span class="p">(</span><span class="s">"⇖"</span> <span class="k">if</span> <span class="n">d</span> <span class="k">else</span> <span class="s">""</span><span class="p">)</span> <span class="o">+</span>\ <span class="p">(</span><span class="s">"⇐"</span> <span class="k">if</span> <span class="n">h</span> <span class="k">else</span> <span class="s">""</span><span class="p">)</span> <span class="n">dist</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">D</span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">])</span> <span class="n">cell_str</span> <span class="o">=</span> <span class="s">"{direction} {star}{dist}{star}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span> <span class="n">direction</span><span class="o">=</span><span class="n">direction</span><span class="p">,</span> <span class="n">star</span><span class="o">=</span><span class="s">" *"</span><span class="p">[((</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">)</span> <span class="ow">in</span> <span class="n">bt</span><span class="p">)],</span> <span class="n">dist</span><span class="o">=</span><span class="n">dist</span><span class="p">)</span> <span class="n">row</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">cell_str</span><span class="p">)</span> <span class="n">table</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> <span class="k">return</span> <span class="n">table</span><span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> <p>Now we are ready to compute the minimum edit distance table, backtrace and alignment. Note that the “#+ATTR_HTML” print statements are there to format the table for this website, they don’t serve any other mysterious purpose.</p> <p>What’s the minimum edit distance between “spell” and “hello”?</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</pre></td><td class="code"><pre><span class="n">word_1</span> <span class="o">=</span> <span class="s">"spell"</span> <span class="n">word_2</span> <span class="o">=</span> <span class="s">"hello"</span> <span class="n">D</span><span class="p">,</span> <span class="n">B</span> <span class="o">=</span> <span class="n">wagner_fischer</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">word_2</span><span class="p">)</span> <span class="n">bt</span> <span class="o">=</span> <span class="n">naive_backtrace</span><span class="p">(</span><span class="n">B</span><span class="p">)</span> <span class="n">edit_distance_table</span> <span class="o">=</span> <span class="n">make_table</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">word_2</span><span class="p">,</span> <span class="n">D</span><span class="p">,</span> <span class="n">B</span><span class="p">,</span> <span class="n">bt</span><span class="p">)</span> <span class="n">alignment_table</span> <span class="o">=</span> <span class="n">align</span><span class="p">(</span><span class="n">word_1</span><span class="p">,</span> <span class="n">word_2</span><span class="p">,</span> <span class="n">bt</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">"Minimum edit distance with backtrace:"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="s">"#+ATTR_HTML: :border 2 :rules all :frame border "</span> <span class="s">":style text-align: right"</span><span class="p">)</span> <span class="c"># org-babel export html properties</span> <span class="k">print</span><span class="p">(</span><span class="n">tb</span><span class="o">.</span><span class="n">tabulate</span><span class="p">(</span><span class="n">edit_distance_table</span><span class="p">,</span> <span class="n">stralign</span><span class="o">=</span><span class="s">"right"</span><span class="p">,</span> <span class="n">tablefmt</span><span class="o">=</span><span class="s">"orgtbl"</span><span class="p">))</span> <span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">Alignment:"</span><span class="p">)</span> <span class="k">print</span><span class="p">(</span><span class="n">tb</span><span class="o">.</span><span class="n">tabulate</span><span class="p">(</span><span class="n">alignment_table</span><span class="p">,</span> <span class="n">tablefmt</span><span class="o">=</span><span class="s">"orgtbl"</span><span class="p">))</span><span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> <p>Minimum edit distance with backtrace (the bold numbers):</p> <table border="2" cellspacing="0" cellpadding="6" rules="all" frame="border" style="text-align: right"> <colgroup> <col class="org-right" /> <col class="org-right" /> <col class="org-right" /> <col class="org-right" /> <col class="org-right" /> <col class="org-right" /> <col class="org-right" /> </colgroup> <tr> <td class="org-right"> &#xa0; </td> <td class="org-right"> # </td> <td class="org-right"> H </td> <td class="org-right"> E </td> <td class="org-right"> L </td> <td class="org-right"> L </td> <td class="org-right"> O </td> </tr> <tr> <td class="org-right"> # </td> <td class="org-right"> <b></b> </td> <td class="org-right"> ⇐ 1 </td> <td class="org-right"> ⇐ 2 </td> <td class="org-right"> ⇐ 3 </td> <td class="org-right"> ⇐ 4 </td> <td class="org-right"> ⇐ 5 </td> </tr> <tr> <td class="org-right"> S </td> <td class="org-right"> ⇑ <b>1</b> </td> <td class="org-right"> ⇑⇖⇐ 2 </td> <td class="org-right"> ⇑⇖⇐ 3 </td> <td class="org-right"> ⇑⇖⇐ 4 </td> <td class="org-right"> ⇑⇖⇐ 5 </td> <td class="org-right"> ⇑⇖⇐ 6 </td> </tr> <tr> <td class="org-right"> P </td> <td class="org-right"> ⇑ 2 </td> <td class="org-right"> ⇑⇖⇐ <b>3</b> </td> <td class="org-right"> ⇑⇖⇐ 4 </td> <td class="org-right"> ⇑⇖⇐ 5 </td> <td class="org-right"> ⇑⇖⇐ 6 </td> <td class="org-right"> ⇑⇖⇐ 7 </td> </tr> <tr> <td class="org-right"> E </td> <td class="org-right"> ⇑ 3 </td> <td class="org-right"> ⇑⇖⇐ 4 </td> <td class="org-right"> ⇖ <b>3</b> </td> <td class="org-right"> ⇐ 4 </td> <td class="org-right"> ⇐ 5 </td> <td class="org-right"> ⇐ 6 </td> </tr> <tr> <td class="org-right"> L </td> <td class="org-right"> ⇑ 4 </td> <td class="org-right"> ⇑⇖⇐ 5 </td> <td class="org-right"> ⇑ 4 </td> <td class="org-right"> ⇖ <b>3</b> </td> <td class="org-right"> ⇖⇐ 4 </td> <td class="org-right"> ⇐ 5 </td> </tr> <tr> <td class="org-right"> L </td> <td class="org-right"> ⇑ 5 </td> <td class="org-right"> ⇑⇖⇐ 6 </td> <td class="org-right"> ⇑ 5 </td> <td class="org-right"> ⇑⇖ 4 </td> <td class="org-right"> ⇖ <b>3</b> </td> <td class="org-right"> ⇐ <b>4</b> </td> </tr> </table> <p>Alignment:</p> <table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides"> <colgroup> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> <col class="org-left" /> </colgroup> <tr> <td class="org-left"> s </td> <td class="org-left"> p </td> <td class="org-left"> e </td> <td class="org-left"> l </td> <td class="org-left"> l </td> <td class="org-left"> &#xa0; </td> </tr> <tr> <td class="org-left"> &#xa0; </td> <td class="org-left"> h </td> <td class="org-left"> e </td> <td class="org-left"> l </td> <td class="org-left"> l </td> <td class="org-left"> o </td> </tr> <tr> <td class="org-left"> d </td> <td class="org-left"> s </td> <td class="org-left"> &#xa0; </td> <td class="org-left"> &#xa0; </td> <td class="org-left"> &#xa0; </td> <td class="org-left"> i </td> </tr> </table> 2016-01-21T11:50:00+00:00 https://giov.dev/2016/01/on-the-objectivity-of-morality-and-beauty-2.html On the objectivity of Morality and Beauty 2016-01-07T17:05:00+00:00 Giovanni Free thoughts and speculations on Truth, Art and Bees. <figure class="image"> <img class="center with-border" src="https://giov.dev/assets/posts/2016-01-07-on-the-objectivity-of-morality-and-beauty/Flower.jpg" /> <figcaption class="center">A beautiful flower.</figcaption> </figure> <p>Sometime before Christmas, a friend pointed me to a <a href="https://youtu.be/J21QuHrIqXg&quot;">great conversation</a> between Sam Harris and Quantum Computation maven David Deutsch. Their discussion revolves on the ideas presented in the physicist’s book “The beginning of infinity”, seamlessly touching on science, knowledge, progress, the relation between humanity and the universe, and the objectivity of morality and beauty.</p> <p>Yes, that’s right, the objectivity of morality and beauty. That’s why I <strong>had</strong> to read “The beginning of infinity”.</p> <!--more--> <p>The book is composed in equal parts of insight and intelligent speculation. And who doesn’t love insight and intelligent speculation? Here’s my highlights of the book, diving directly into the controversial bits!</p> <h2 id="reality-truth-objectivity">Reality, Truth, Objectivity</h2> <p>Deutsch defines something to be objective if it figures in our best explanations of the world. This way, objectivity doesn’t directly refer to the reality <strong>out there</strong> anymore. That’s because there’s no way to access that reality in its raw form – no such thing as knowledge from raw sensory experience: all knowledge is theory-laden.</p> <p>All we have is our theories. However, our theories <strong>can</strong> and do sometimes contain elements of truth, reflecting the properties of the reality out there. The tools we have to make sure that the content of truths of our theories grows are conjecture and criticism. That is, we improve our theories by conjecturing competing explanations and we criticize them against some criteria. The criteria are theory-laden and are themselves subject to change, again through conjecture and criticism .</p> <p>In physics, for example, we can consider a conjectured theory as an improvement in respect to the others if it better meets the criteria of explaining the available data, do so simply, and of producing testable predictions. These criteria weren’t born with physics itself, but emerged after centuries of discussion and criticism – and are not presently devoid of controversy either.</p> <p>The power of conjecture and criticism – the power of reason itself – can be applied to more than just science. Philosophical theories that can be criticized against <em>factual knowledge</em> are also fair play. Take Morality and Aesthetics, for example.</p> <h2 id="reason-and-morality">Reason and Morality</h2> <p>Deutsch defines moral philosophy as tackling the problem of “what to do next”, that is to decide between a wide spectrum of options by reasoning about which options are better and which are worse. The Moral relativism point of view on the question is that there’s no real “better” or “worse”: these judgments only exist inside the arbitrary standards of culture.</p> <p>Here’s where the <em>factual knowledge</em> in Deutsch’s claim makes its entrance. Moral theories do not just appear out of thin cultural air. They are connected with the physical world through the explanations we create to support them.</p> <p>As an example, a person could consider gay adoption wrong because they think that homosexuality is a severe mental illness and, as a consequence of being raised by mentally ill parents, the child would suffer harm.</p> <p>This moral theory <em>can</em> be pitted against physical reality. In fact, we observe plenty of adoptions by gay couples in which the child develops normally. At the same time, current science excludes the possibility that homosexuality is a severe mental illness, through its own reality-anchored cycle of conjecture and criticism. The explanation which underlies the moral theory is at odds with reality: we need a better moral theory. Luckily, we can create one through reality-anchored conjecture and criticism.</p> <p>Morality is not the only philosophical theory that can be pitted against physical reality. Deutsch considers Aesthetics as another important example</p> <h2 id="beauty-the-common-structure">Beauty, the common structure</h2> <p>Aesthetic pleasure can be derived by a multitude of means, as listening to great composers, looking at great paintings, or studying mathematics and physics.</p> <p>Deutsch believes that there is a common structure underlying what we consider Aesthetically pleasing. If we want to explain <em>why</em> we find some things Aesthetically pleasing but not others, that structure has to be part of our explanation. Thus, given the definition of objectivity presented at the beginning – “objective is that which figures in our best explanations of the world” – the structure has to be considered objective.</p> <p>To explain why he guesses that a common structure exists, Deutsch uses an argument which I really can’t get myself to like. He thinks that there’s no good explanation yet for why we like flowers, and guesses that the reason is that flowers evolved to produce objective beauty, and we have some of the knowledge to recognize it. In his argument, flowers and bees co-evolved so that the flowers could produce (and the bees recognize) a code to signal past the genetic rift between them – a code with universal reach. Humans are also separated by rifts, due to the fact that the content of each human mind is radically different from that of the others. For this reason human artists produced knowledge of the same nature as the one contained in the flowers’ genes, knowledge about what we call Beauty – a common structure which can inform codes of universal reach.</p> <p>This is certainly a piece of intelligent speculation, but not very insightful to me. The flowers argument is a way for Deutsch to anchor Aesthetics to factual knowledge about the physical world, and detach it from parochial human experiences: if we are not the only thing in the universe able to produce and recognize beauty, then beauty cannot be cultural – flowers and bees don’t participate in our culture. Still, this argument is not really convincing to me. Is the rift between humans really as profound as that between flowers and bees? Humans can create profound connections even when they cannot even rely on a common culture or a common language to communicate; is art then really answering a need for universal communication? And why should codes with universal reach be biased to go towards beauty anyway? Would humans which never had contact with art not consider flowers beautiful? Given that bees evolved to (imperfectly) recognize beauty, and we created knowledge to (imperfectly) produce beauty, does that mean that bees would be attracted to our art if we adapted it to fit their way of experiencing the world?</p> <p>I think we are in need of a better argument if we want to introduce Aesthetics to the physical world.</p> <h2 id="conclusions">Conclusions</h2> <p>I absolutely loved “The beginning of infinity”. Deutsch’s views on the nature of knowledge and the means to attain it resonate with my own – never been a fan of positivism nor relativism. I think he is spot-on when he writes about science and how it is about better and better explanations rather than just predictions.</p> <p>I still have to think carefully about his views about morality, as they imply that different cultures applying conjecture and criticism to theories, and their moral theories in particular, would in the limit converge towards the same moral theory. Then the aim of moral philosophy becomes to try and approximate a singular and specific <em>moral reality</em>. Can such a thing really exist?</p> 2016-01-07T17:05:00+00:00 https://giov.dev/2015/11/heteroclinic-switching-simulator-and-visualizer.html Heteroclinic Switching simulator and visualizer 2015-11-15T16:53:26+00:00 Giovanni Switching dynamics, heteroclinic people and nice visualizations. <p>I’ll soon be pleased to give a talk about my work on heteroclinic dynamics to other heteroclinic people with heteroclinic interests at the <em>Heteroclinic dynamics in neuroscience</em> workshop this December in Nice. In this heteroclinic setting, I thought it would be worth it to have a simple interactive simulation to play with while I illustrate my findings, as I don’t want the talk to be boring, and nothing boosts understanding like a good visualization!</p> <!--more--> <p>So I spent the last few days programming a simple visualizer in Python for the switching dynamics in the system described <a href="http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.109.018701">here</a>, built upon the simulator I am using to produce the Terabyte of data needed for my next paper with <a href="https://www.researchgate.net/profile/Fabio_Schittler_Neves">Fabio Neves</a>. <em>Ta-da</em>:</p> <p>In the meantime, I thought it would be nice to put everything on GitHub, so maybe other heteroclinic people can give it a heteroclinic try. You never know!</p> <p>So, <a href="https://github.com/TuringMachinegun/heteroclinic_simulator">here it is</a>! At the moment what it can do is to simulate the system I linked above, but I programmed it to be easily generalizable if needed. That is, if I find that somebody is actually interested in a general version – at the moment it contains bits that make it special-purpose.</p> <p>Otherwise, it will just be an exercise in reproducible and open research, which I am happy with anyway.</p> 2015-11-15T16:53:26+00:00 https://giov.dev/2015/07/putting-some-make-up-on-my-org-mode-flashcards.html Putting some make up on my org-mode flashcards 2015-07-16T00:00:00+00:00 GC If org-mode had running water and a bed, I'd live in it. Some tweaks to org-drill to get the best flashcards experience Lisp has to offer. <p>Lately I’ve been playing with <a href="https://bitbucket.org/eeeickythump/org-drill">org-drill</a>, an extension to Emacs org-mode implementing a spaced repetition algorithm for flashcard drills. That is, org-drill lets you write flashcards in the form of org outlines plus  special syntax, and have study sessions where the cards are presented to you using some <a href="https://en.wikipedia.org/wiki/Spaced_repetition">special algorithms</a> that should improve retention. The flashcards can have hidden text/images, present hints, have multiple faces, together with other useful settings.</p> <!--more--> <div id="attachment_1068" style="width: 617px" class="wp-caption aligncenter"> <a href="http://giov.dev/images/ol_fc_small.gif"><img class="wp-image-1068 size-full" src="http://giov.dev/images/ol_fc_small.gif" alt="org-drill_flashcard" width="607" height="291" data-wp-pid="1068" /></a> <p class="wp-caption-text"> Example org-drill flashcard. </p> </div> <p>One of the main points of org-drill is of course its integration in org-mode, letting you keep your flashcards together with your other plain-text notes. The flashcards are no different from any other org outline, apart from some special properties which org-drill saves in the :PROPERTIES: drawer, and some syntax to indicate parts of text that should be hidden in the flashcard (to prompt a recall on the side of the student), and hints (to help the recall).</p> <p>I find org-drill very useful to have quick sessions to strengthen key facts from what I’m studying. But I wanted to integrate it better with my org-mode workflow. That is, I wanted the key facts that I keep in my notes to be treated as flashcards when needed, and as normal notes otherwise. The org-drill clozes (with my customized delimiters) look something like this:</p> <ul style="list-style-type: circle;"> <li> The answer to the ultimate question of life, the universe and everything is !| 42 || six by nine |! ,  as computed by !| Deep Thought |! </li> </ul> <p>If you are using org-drill and didn’t customize the delimiters, you’ll have [ and ] instead of !| and |! . You can probably already see that the !| fact || hint |! syntax can be a little annoying when reading the notes.</p> <p>To solve this problem I’ve written some functions that get the special syntax out of the way when exporting the org-mode file, and when reading the notes as notes – not as flashcards. Elisp to the rescue!</p> <p>Here’s a before-after comparison:</p> <div id="attachment_1080" style="width: 576px" class="wp-caption aligncenter"> <a href="http://giov.dev/images/hideshow1.gif"><img class="wp-image-1080 size-full" src="http://giov.dev/images/hideshow1.gif" alt="" width="566" height="386" /></a> <p class="wp-caption-text"> An example org file with and without the org-drill special syntax. </p> </div> <p>Much neater! As a plus, the special syntax is also hidden when viewing flash-cards, making them neater as well:</p> <div id="attachment_1082" style="width: 617px" class="wp-caption aligncenter"> <a href="http://giov.dev/images/new_fc.gif"><img class="wp-image-1082 size-full" src="http://giov.dev/images/new_fc.gif" alt="org-drill_flashcard_no_delimiters" width="607" height="291" /></a> <p class="wp-caption-text"> Org-drill flashcard with no delimiters in visible clozes. </p> </div> <p> </p> <p>And the export works just as well:</p> <div id="attachment_1084" style="width: 531px" class="wp-caption aligncenter"> <a href="http://giov.dev/images/refile.png"><img class="wp-image-1084 size-full" src="http://giov.dev/images/refile.png" alt="pdf export" width="521" height="391" sizes="(max-width: 521px) 100vw, 521px" /></a> <p class="wp-caption-text"> Snapshot of the exported pdf. </p> </div> <p>I’m pretty happy with the result. It is a small thing but it makes a huge difference in that I can just write my notes as I normally do, very quickly make them flashcards if I need to, and seamlessly go back and forth between the two representations depending on the way I’m studying.</p> <p>I’d like to push this further, as this little trick only works when my notes are a sequence of distinct and well-organized little pieces of information. This happens to be the case for the probability notes I used as an example here, but it needn’t always be. Sometimes it makes more sense to maybe have long-running paragraphs in a more discursive form, depending on the type of information the notes are capturing. It would be great to use a special syntax to create flashcards from tokenized concepts inside the paragraph, and hide/show that as well when need be. Who knows, maybe this will be a side project I’ll actually manage to work on!</p> <p>Anyway, here is the small section of code accomplishing what I’ve described (and also my first attempt to write non-trivial emacs lisp… I hope I’m doing it the right way). Note that this code doesn’t assume the use of my custom delimiters, the default delimiters are supported. From the countless times I stole elisp snippets from someone’s blog, I know that there’s somebody out there that will appreciate this:</p> <figure class="highlight"><pre><code class="language-emacs-lisp" data-lang="emacs-lisp"><table style="border-spacing: 0"><tbody><tr><td class="gutter gl" style="text-align: right"><pre class="lineno">1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66</pre></td><td class="code"><pre><span class="p">(</span><span class="nb">require</span> <span class="ss">'org-drill</span><span class="p">)</span> <span class="c1">;; remove clozes when exporting ;;</span> <span class="p">(</span><span class="nb">defun</span> <span class="nv">gsc/drill-compute-cloze-regexp</span> <span class="p">()</span> <span class="s">"Same regular expression as in org-drill-cloze-regexp, but adding a group for the first delimiter, so that it can be distinguished easily in a match."</span> <span class="p">(</span><span class="nv">concat</span> <span class="s">"\$$"</span> <span class="p">(</span><span class="nv">regexp-quote</span> <span class="nv">org-drill-left-cloze-delimiter</span><span class="p">)</span> <span class="s">"\$$\$$[[:cntrl:][:graph:][:space:]]+?\$$\$$\\|"</span> <span class="p">(</span><span class="nv">regexp-quote</span> <span class="nv">org-drill-hint-separator</span><span class="p">)</span> <span class="s">".+?\$$\$$"</span> <span class="p">(</span><span class="nv">regexp-quote</span> <span class="nv">org-drill-right-cloze-delimiter</span><span class="p">)</span> <span class="s">"\$$"</span><span class="p">))</span> <span class="p">(</span><span class="nb">defun</span> <span class="nv">gsc/drill-cloze-removal</span> <span class="p">(</span><span class="nv">backend</span><span class="p">)</span> <span class="s">"Remove drill clozes in the current buffer. BACKEND is the export back-end being used, as a symbol."</span> <span class="p">(</span><span class="nv">while</span> <span class="p">(</span><span class="nv">re-search-forward</span> <span class="p">(</span><span class="nv">gsc/drill-compute-cloze-regexp</span><span class="p">)</span> <span class="no">nil</span> <span class="no">t</span><span class="p">)</span> <span class="c1">;; (Copy-pasted this from org-drill-el)</span> <span class="c1">;; Don't delete:</span> <span class="c1">;; - org links, partly because they might contain inline</span> <span class="c1">;; images which we want to keep visible.</span> <span class="c1">;; - LaTeX math fragments</span> <span class="c1">;; - the contents of SRC blocks</span> <span class="p">(</span><span class="nb">unless</span> <span class="p">(</span><span class="nv">save-match-data</span> <span class="p">(</span><span class="nb">or</span> <span class="p">(</span><span class="nv">org-pos-in-regexp</span> <span class="p">(</span><span class="nv">match-beginning</span> <span class="mi">0</span><span class="p">)</span> <span class="nv">org-bracket-link-regexp</span> <span class="mi">1</span><span class="p">)</span> <span class="p">(</span><span class="nv">org-in-src-block-p</span><span class="p">)</span> <span class="p">(</span><span class="nv">org-inside-LaTeX-fragment-p</span><span class="p">)))</span> <span class="p">(</span><span class="nv">replace-match</span> <span class="s">"\\2"</span> <span class="no">nil</span> <span class="no">nil</span><span class="p">))))</span> <span class="p">(</span><span class="nv">add-hook</span> <span class="ss">'org-export-before-processing-hook</span> <span class="ss">'gsc/drill-cloze-removal</span><span class="p">)</span> <span class="c1">;; hide clozes in text ;;</span> <span class="p">(</span><span class="nb">defvar</span> <span class="nv">gsc/drill-groups-to-hide</span> <span class="o">'</span><span class="p">(</span><span class="mi">1</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">)</span> <span class="s">"Group 1 and 4 are the left and right delimiters respectively, group 3 is the cloze hint."</span><span class="p">)</span> <span class="p">(</span><span class="nv">setplist</span> <span class="ss">'gsc/inv-cloze</span> <span class="o">'</span><span class="p">(</span><span class="nv">invisible</span> <span class="no">t</span><span class="p">))</span> <span class="p">(</span><span class="nb">defun</span> <span class="nv">gsc/hide-clozes-groups</span> <span class="p">()</span> <span class="p">(</span><span class="nv">save-excursion</span> <span class="p">(</span><span class="nv">goto-char</span> <span class="p">(</span><span class="nv">point-min</span><span class="p">))</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nv">cloze-regexp</span> <span class="p">(</span><span class="nv">gsc/drill-compute-cloze-regexp</span><span class="p">)))</span> <span class="p">(</span><span class="nv">while</span> <span class="p">(</span><span class="nv">re-search-forward</span> <span class="nv">cloze-regexp</span> <span class="no">nil</span> <span class="no">t</span><span class="p">)</span> <span class="p">(</span><span class="nb">loop</span> <span class="nv">for</span> <span class="nv">group</span> <span class="nv">in</span> <span class="nv">gsc/drill-groups-to-hide</span> <span class="nb">do</span> <span class="p">(</span><span class="nv">overlay-put</span> <span class="p">(</span><span class="nv">make-overlay</span> <span class="p">(</span><span class="nv">match-beginning</span> <span class="nv">group</span><span class="p">)</span> <span class="p">(</span><span class="nv">match-end</span> <span class="nv">group</span><span class="p">))</span> <span class="ss">'category</span> <span class="ss">'gsc/inv-cloze</span><span class="p">))))))</span> <span class="p">(</span><span class="nb">defun</span> <span class="nv">gsc/show-clozes-all</span> <span class="p">()</span> <span class="p">(</span><span class="nv">save-excursion</span> <span class="p">(</span><span class="nv">goto-char</span> <span class="p">(</span><span class="nv">point-min</span><span class="p">))</span> <span class="p">(</span><span class="nv">while</span> <span class="p">(</span><span class="nv">re-search-forward</span> <span class="p">(</span><span class="nv">gsc/drill-compute-cloze-regexp</span><span class="p">)</span> <span class="no">nil</span> <span class="no">t</span><span class="p">)</span> <span class="p">(</span><span class="nv">remove-overlays</span> <span class="p">(</span><span class="nv">match-beginning</span> <span class="mi">1</span><span class="p">)</span> <span class="p">(</span><span class="nv">match-end</span> <span class="mi">4</span><span class="p">)</span> <span class="ss">'category</span> <span class="ss">'gsc/inv-cloze</span><span class="p">))))</span> <span class="p">(</span><span class="nb">defun</span> <span class="nv">gsc/hide-show-clozes</span> <span class="p">(</span><span class="nv">arg</span><span class="p">)</span> <span class="s">"If called with no argument, hides delimiters and hints for org-drill clozes. If called with the C-u universal argument, it shows them."</span> <span class="p">(</span><span class="nv">interactive</span> <span class="s">"p"</span><span class="p">)</span> <span class="p">(</span><span class="nb">case</span> <span class="nv">arg</span> <span class="p">(</span><span class="mi">1</span> <span class="p">(</span><span class="nv">gsc/hide-clozes-groups</span><span class="p">))</span> <span class="p">(</span><span class="mi">4</span> <span class="p">(</span><span class="nv">gsc/show-clozes-all</span><span class="p">))))</span><span class="w"> </span></pre></td></tr></tbody></table></code></pre></figure> 2015-07-16T00:00:00+00:00 https://giov.dev/2015/03/floating-point-representation-visualizer.html Floating point representation visualizer 2015-03-30T16:45:09+00:00 Giovanni <p>Hi everyone,</p> <p>I’ve been teaching an introductory course on the Theory of Computation at Plymouth University. The topic of my last <a href="http://nbviewer.ipython.org/github/TuringMachinegun/float_visualizer/blob/master/visualizer.ipynb" target="_blank">lecture</a> was the representation of real numbers in computers, and the unavoidable errors introduced by any choice of representation. Particularly, the lecture was focused on the floating point representation of real numbers.</p> <!--more--> <p>To make the nature of the float representation more intuitive to the students, I programmed a little visualizer which plots the numbers that can be represented with a floating point representation, given the number of bits in the exponential and in the mantissa. Within the visualizer, it is possible to change the number of bits in the exponential and the mantissa and observe how the set of representable numbers change.</p> <p>It is also possible to sum two numbers, <strong>a</strong> and <strong>b</strong>, and observe how errors come to be and get the visual intuition of why they are unavoidable (hopefully).</p> <p>– <a href="https://github.com/TuringMachinegun/float_visualizer/blob/master/visualizer.ipynb" target="_blank">Here</a>, the ipython notebook I used during my lecture.</p> <p>– <a href="https://github.com/TuringMachinegun/float_visualizer/raw/master/visualizer.zip" target="_blank">Here</a>, the zipped executable for the visualizer.</p> <p>– <a href="https://github.com/TuringMachinegun/float_visualizer/blob/master/visualizer.py" target="_blank">Here</a> the python code (you will need python 2.7, with matplotlib and numpy libraries).</p> <p><em>At the moment of writing the visualizer is still lacking some features. In particular, I’d like to make the representation more IEEE-754 compliant by adding support for signed 0’s, signed infinity, and NaN values. I’ll do that soon, but for the moment, I think the visualizer is already useful as it is.</em></p> <p> </p> 2015-03-30T16:45:09+00:00 https://giov.dev/2014/07/bat-navigator-prototype.html Bat Navigator prototype 2014-07-18T16:48:00+00:00 Giovanni The Bat Navigator is finally here! <p>Hi everyone,</p> <p>from the last post, I added a third ultrasonic sensor to the project.<br /> The purpose of this sensor is to detect obstacles like tables or chairs, that are too low to be detected by the other two sensors. If an obstacle is detected, then the pitch of the two piezo transducers goes down. At the moment I implemented it as an all-or-none detection, so the user doesn’t actually know how high the obstacle is, just that it is there.</p> <!--more--> <p>I want to change it later so that the new central sensor has its own beeping using the two piezos, so I won’t post the code here, as it is just a temporary fix. The reason why I went with a temporary fix is that I actually couldn’t wait to build and test the glasses, see if they actually worked, and see how well they worked.</p> <p>Today I built the glasses.</p> <p>So, here I present you the Bat Navigator version 0.1! <img src="https://giov.dev/images/IMG_20140718_163122.jpg" alt="Project material" caption="Bat Navigator" /></p> <p><strong>Some close-ups of the glasses!</strong></p> <div class="row"> <div class="large-4 columns"> <a href="https://giov.dev/images/IMG_20140718_161831.jpg"><img src="https://giov.dev/images/IMG_20140718_161831.jpg" /></a> </div> <div class="large-4 columns"> <a href="https://giov.dev/images/IMG_20140718_161802.jpg"><img src="https://giov.dev/images/IMG_20140718_161802.jpg" alt="Notice the flexible tube which permits the regulation of the central sensor angle, idea from Valerio Biscione. Flexible tube scavenged from an innocent reading light." /></a> </div> <div class="large-4 columns"> <a href="https://giov.dev/images/IMG_20140718_161810.jpg"><img src="https://giov.dev/images/IMG_20140718_161810.jpg" alt="I made some custom connectors using Epoxy and ferrules (i.e. stuff I’ve found at the local electronics store)" /></a> </div> </div> <p>Nice, right? Me and my friend <a href="http://valerio-biscione.psychology-search.com/">Valerio</a> tested them out today.</p> <p>Our testing procedure consisted of the following steps:</p> <ul> <li>One of us wears the glasses, and closes his eyes.</li> <li>The other one spins him around a few times.</li> <li>The one who wears the glasses tries to walk around the building without hurting himself.</li> <li>The other one makes sure that he succeeds in not dying falling down some stairs</li> <li>A <a href="http://robinread.me/">third person</a> plays pranks on the one with the eyes closed (essential part of testing)</li> </ul> <p>The Bat Navigator actually works much better than I expected, I was able to navigate the building, even if at a really slow pace. I got the hang of it almost immediately, recognizing walls and their orientation, corridors, and tables. I’m pretty confident that if I keep working on it, it could actually become a feasible way to navigate indoor environments without using eyesight.</p> <p>It will be a while before I update the blog again, as the next steps will either involve adding another couple of sensors to the side and figure out a way to convey that information without producing information overload for the user, or building a second prototype where everything is mounted on the glasses (so something that can actually be used normally). Both of the options require a good amount of work, and I won’t have that kind of time in the near future.<br /> Maybe I will continue working on this from the second half of August, but, until then, I’ll have to focus on my central projects.</p> <p>In the mean time, I want to thank Ricardo for giving me precious advice on how to go about building this prototype, and Valerio for his many tips, ideas and help in testing the Bat Navigator.</p> <p>Bye bye now, I’ll keep you posted! (In a while)</p> <p>TM</p> 2014-07-18T16:48:00+00:00 https://giov.dev/2014/07/lets-dance-to-the-sound-of-piezos.html Let's dance to the sound of piezos 2014-07-14T15:43:00+00:00 Giovanni <p>Hi everyone.</p> <p>As I wrote in the last post, I found out that vibration motors do not have much “expressive power”, not as much as I thought anyway. For this reason, I am switching to piezoelectric transducers, which offer a greater range of expression, are easy to control and wire up, and consume almost nothing.</p> <!--more--> <p>At first, I wanted the piezos to produce a fixed pitch, and vary the volume of the sound as a function of the distance of the obstacles detected by the ultrasound sensors. Turns out, it is not that easy to do so with a piezoelectric transducer, <a href="http://www.electro-tech-online.com/threads/transducer-piezo-is-it-a-buzzer.3463/#post-16452">as it would be with a buzzer.</a></p> <p>Then I tried to vary the pitch of the produced sounds as a function of the distance of the obstacles, but I didn’t like the result. I thought it would be quite annoying to listen to this continuous change of pitch while using the glasses for some time. The good thing is that it worked quite well as a theremin!</p> <div style="clear: both; text-align: center;"> </div> <div style="text-align: center;"> <span style="font-size: x-small;">(Sorry about the noise, but piezo transducers produce very low sounds)</span> </div> <p>In the end, I decided to keep volume and pitch fixed, and vary the beeping rhythm, like in parking sensors:</p> <div style="clear: both; text-align: center;"> </div> <p>The circuit here is quite simple – the mess you can see in the video is actually just because I was too lazy to tidy everything up. Each ultrasonic sensor is connected to Ground, to the 5V pin, and to a single digital pin (one for each sensor). In fact, with the NewPing library, you can use a single pin to ping the sensor, and to receive the reading. <br /> The piezo transducers were just connected each to a digital pin and to ground. I think I will add some resistance to adjust the volume later on, and maybe a linear potentiometer to control the volume manually.</p> <p>Now I can guide you through the code I’ve used here. <br /> I’ll explain the key functions and then show the complete code, I’m assuming you already know the basics for C coding.</p> <pre>void loop(){ curr_millis=millis(); //take time passed from arduino start // if a millisecond has passed if (curr_millis != prev_millis){ debugprint("starting."); debugprintln(curr_millis); debugprint("\n"); // ping ultrasound sensors and retrieve distances ping_us_sensors(curr_millis); l_read_array[reading_n] = distances; //reading_n is incremented only when both sensors have been pinged r_read_array[reading_n] = distances; debugprint("now returning to loop \n"); debugprint("distances are: \n"); debugprintln(distances); debugprintln(distances); // average last READINGS readings int l_sum = 0; int r_sum = 0; for (int i=0; i&lt;READINGS; i++){ l_sum += l_read_array[i]; r_sum += r_read_array[i]; } float l_ave_distance = l_sum/float(READINGS) - MIN_DISTANCE; float r_ave_distance = r_sum/float(READINGS) - MIN_DISTANCE; if(l_ave_distance&lt;0) l_ave_distance=0; // make sure not to have negative distances if(r_ave_distance&lt;0) r_ave_distance=0; debugprint("and average distances:\n"); debugprintln(r_ave_distance); debugprintln(l_ave_distance); debugprint("\n"); // make piezos beep depending on the distances of the obstacles pulse_piezos(curr_millis, l_ave_distance, r_ave_distance); prev_millis = curr_millis; } } </pre> <p>This is the function which is called continuously as the Arduino is turned on. What it does is: <br /> – Ping the US sensors. <br /> – Average out the last READINGS readings, where READING is a number defined before. <br /> – Shift the distance down of MIN_DISTANCE. This is used so that the piezos can already beep at the highest speed when an obstacle is presented at MIN_DISTANCE. <br /> – Make the piezos beep. <br /> The first and last actions in this list are actually carried out by two functions which are external to the loop. Let’s have a look at the function which pings the sensors.</p> <pre>// ping the us sensors and retrieve the distances from obstacles void ping_us_sensors(long curr_millis){ debugprint("Entered ping_us_sensor fn\n"); debugprint("time passed from last ping:"); debugprintln(curr_millis - last_ping_time); // time in microseconds it takes for the ultrasonic sound wave // to travel from the ultrasonic sensor, hit an obstacle, and return unsigned int uS; // some time must pass between a ping and the next one if(curr_millis - last_ping_time &gt;= PING_INTERVAL){ // if readings_n is equal to READINGS -&gt; readings_n = 0 // used to populate reading arrays reading_n%=READINGS; // used to ping sequentially all US sensors (in this case, 2) pinger_id++; pinger_id %= US_NUM; debugprint("Pinging sensor number \n"); debugprintln(pinger_id); uS = sonar[pinger_id].ping(); last_ping_time = curr_millis; // compute distance distances[pinger_id] = uS/US_ROUNDTRIP_CM; // when the distance is too much, the sensor sends a 0 // better to just set that to MAX_DISTANCE if (distances[pinger_id]==0) distances[pinger_id]=MAX_DISTANCE; debugprint("Distance: \n"); debugprintln(distances[pinger_id]); if(pinger_id%US_NUM == 0)reading_n++; } debugprint("\n"); } </pre> <p>This function pings the sensors one at a time, retrieving the distance for each one. To do that, it makes sure that enough time has passed since the last ping (PING_INTERVAL). <br /> Also, it both the US have been pinged, it updates the reading_n variable. That variable is used in the loop function to populate the reading arrays.</p> <p>Now for the function that makes the piezos beep.</p> <pre>// This function makes the piezo beep based on the distance of the obstacles void beep_piezos(long curr_millis, long l_ave_distance, long r_ave_distance) { int r_interval, l_interval; // from the distance, assign beep intervals for the piezo transducers l_interval = int(l_ave_distance*DIST_TO_BEEP_MULTIPLIER); r_interval = int(r_ave_distance*DIST_TO_BEEP_MULTIPLIER); debugprint("entering beep_piezos fn.\n"); debugprint("time passed from last right beep:"); debugprintln(curr_millis - r_last_beep); debugprint("time passed from last left beep:"); debugprintln(curr_millis - l_last_beep); // if enough time has passed since the last beep of the right piezo, // a new beep is started in the right piezo if(curr_millis - r_last_beep &gt; r_interval){ debugprint("right beep sent\n"); r_last_beep = curr_millis; } // controls the actual wave that constitues the beep if(curr_millis - r_last_beep &lt; BEEP_DURATION){ if(r_phase == R_BEEP_LOW_RATIO){ digitalWrite(R_PIEZO, HIGH); r_phase=0; } else{ digitalWrite(R_PIEZO,LOW); r_phase++; } } // if enough time has passed since the last beep of the left piezo, // a new beep is started in the left piezo if(curr_millis - l_last_beep &gt; l_interval){ debugprint("left beep sent\n"); l_last_beep = curr_millis; } // controls the actual wave that constitues the beep if(curr_millis - l_last_beep &lt; BEEP_DURATION){ if(l_phase == L_BEEP_LOW_RATIO){ digitalWrite(L_PIEZO, HIGH); l_phase=0; } else{ digitalWrite(L_PIEZO,LOW); l_phase++; } } debugprint("\n"); } </pre> <p>This function makes the piezo beep more or less frequently depending on the distance of the left and right obstacles. <br /> When the right time interval has passed, the function starts a beep. A beep here is a square wave sent to a piezo for a certain PULSE_DURATION time.</p> <p>The pitch of the beep sound is controlled by the R and L_BEEP_LOW_RATIO. Given that this function is called by the loop every 1 ms, if for example L_BEEP_LOW_RATIO = 2, then a series of HIGH – LOW – LOW is sent, where each signal lasts a millisecond. <br /> The frequency of the resulting square wave is 1/3 of 1000 Hz (again, because the function is called every millisecond, 1000 times a second).</p> <p>Finally the complete code, hopefully it should be easy enough now to read through.</p> <pre>#include // pins #define R_US 11 // right ultrasonic sensor pin #define L_US 8 // left ultrasonic sensor pin #define R_PIEZO 3 // right piezo pin #define L_PIEZO 5 // left piezo pin // parameters #define US_NUM 2 // total number of US sensors #define READINGS 4 // US readings to average #define MIN_DISTANCE 20 // distance in cm which make the piezo beep like crazy #define MAX_DISTANCE 300 // maximum distance for a valid reading from the US sensors #define PING_INTERVAL 40 // interval between pings #define BEEP_DURATION 30 // how much a beep lasts in ms #define DIST_TO_BEEP_MULTIPLIER 5 // to adjust the beeping relative to distance #define L_BEEP_LOW_RATIO 2 // this regulates the frequency of the note for the left piezo #define R_BEEP_LOW_RATIO 1 // // the next lines turn debugging on or off //#define DEBUG #ifdef DEBUG #define debugbegin(x) Serial.begin(x) #define debugprint(x) Serial.print(x) #define debugprintln(x) Serial.println(x) #else #define debugbegin(x) #define debugprint(x) #define debugprintln(x) #endif /*DEBUG*/ NewPing sonar[US_NUM] = { NewPing(R_US, R_US, MAX_DISTANCE), // NewPing setup. NewPing(L_US, L_US, MAX_DISTANCE) // NewPing setup. }; // variables used to keep track of time long curr_millis=0; long prev_millis=0; // variables used to keep track of the phase of the square wave for each piezo int r_phase = 0; int l_phase = 0; // keep track of the last time the piezos beeped long r_last_beep = 0; long l_last_beep = 0; int reading_n = 0; // used to update l_read_array/right arrays int pinger_id = 0; // used to alternate pinging between sensors int l_read_array[READINGS]; // used to store and average readings int r_read_array[READINGS]; long last_ping_time = 0; // last time a US was pinged int distances[US_NUM]; // stores the distances retrieved by the US sensors // This function makes the piezo beep based on the distance of the obstacles void beep_piezos(long curr_millis, long l_ave_distance, long r_ave_distance) { int r_interval, l_interval; // from the distance, assign beep intervals for the piezo transducers l_interval = int(l_ave_distance*DIST_TO_BEEP_MULTIPLIER); r_interval = int(r_ave_distance*DIST_TO_BEEP_MULTIPLIER); debugprint("entering beep_piezos fn.\n"); debugprint("time passed from last right beep:"); debugprintln(curr_millis - r_last_beep); debugprint("time passed from last left beep:"); debugprintln(curr_millis - l_last_beep); // if enough time has passed since the last beep of the right piezo, // a new beep is started in the right piezo if(curr_millis - r_last_beep &gt; r_interval){ debugprint("right beep sent\n"); r_last_beep = curr_millis; } // controls the actual wave that constitues the beep if(curr_millis - r_last_beep &lt; BEEP_DURATION){ if(r_phase == R_BEEP_LOW_RATIO){ digitalWrite(R_PIEZO, HIGH); r_phase=0; } else{ digitalWrite(R_PIEZO,LOW); r_phase++; } } // if enough time has passed since the last beep of the left piezo, // a new beep is started in the left piezo if(curr_millis - l_last_beep &gt; l_interval){ debugprint("left beep sent\n"); l_last_beep = curr_millis; } // controls the actual wave that constitues the beep if(curr_millis - l_last_beep &lt; BEEP_DURATION){ if(l_phase == L_BEEP_LOW_RATIO){ digitalWrite(L_PIEZO, HIGH); l_phase=0; } else{ digitalWrite(L_PIEZO,LOW); l_phase++; } } debugprint("\n"); } // ping the us sensors and retrieve the distances from obstacles void ping_us_sensors(long curr_millis){ debugprint("Entered ping_us_sensor fn\n"); debugprint("time passed from last ping:"); debugprintln(curr_millis - last_ping_time); // time in microseconds it takes for the ultrasonic sound wave // to travel from the ultrasonic sensor, hit an obstacle, and return unsigned int uS; // some time must pass between a ping and the next one if(curr_millis - last_ping_time &gt;= PING_INTERVAL){ reading_n%=READINGS; //used to populate the distance arrays // used to ping sequentially all US sensors (in this case, 2) pinger_id++; pinger_id %= US_NUM; debugprint("Pinging sensor number \n"); debugprintln(pinger_id); uS = sonar[pinger_id].ping(); last_ping_time = curr_millis; // compute distance distances[pinger_id] = uS/US_ROUNDTRIP_CM; // when the distance is too much, the sensor sends a 0 // better to just set that to MAX_DISTANCE if (distances[pinger_id]==0) distances[pinger_id]=MAX_DISTANCE; debugprint("Distance: \n"); debugprintln(distances[pinger_id]); if(pinger_id%US_NUM == 0)reading_n++; } debugprint("\n"); } void setup() { debugbegin(115200); // set piezo pins to OUTPUT pinMode(R_PIEZO,OUTPUT); pinMode(L_PIEZO,OUTPUT); //initializing the arrays for (int i=0; i&lt;READINGS; i++){ l_read_array[i] = 0; r_read_array[i] = 0; } } void loop(){ curr_millis=millis(); //take time passed from arduino start // if a millisecond has passed if (curr_millis != prev_millis){ debugprint("starting."); debugprintln(curr_millis); debugprint("\n"); // ping ultrasound sensors and retrieve distances ping_us_sensors(curr_millis); l_read_array[reading_n] = distances; //reading_n is incremented only when both sensors have been pinged r_read_array[reading_n] = distances; debugprint("now returning to loop \n"); debugprint("distances are: \n"); debugprintln(distances); debugprintln(distances); // average last READINGS readings int l_sum = 0; int r_sum = 0; for (int i=0; i&lt;READINGS; i++){ l_sum += l_read_array[i]; r_sum += r_read_array[i]; } float l_ave_distance = l_sum/float(READINGS) - MIN_DISTANCE; float r_ave_distance = r_sum/float(READINGS) - MIN_DISTANCE; if(l_ave_distance&lt;0) l_ave_distance=0; // make sure not to have negative distances if(r_ave_distance&lt;0) r_ave_distance=0; debugprint("and average distances:\n"); debugprintln(r_ave_distance); debugprintln(l_ave_distance); debugprint("\n"); // make piezos beep depending on the distances of the obstacles beep_piezos(curr_millis, l_ave_distance, r_ave_distance); prev_millis = curr_millis; } } </pre> <p>OK, that’s it for this post, hope you enjoyed it! For the next post, I will probably implement a third sensor to sense low-frontal objects (but I want to use only two piezo transducers).</p> <p>I’ll keep you posted!</p> 2014-07-14T15:43:00+00:00 https://giov.dev/2014/06/adding-the-ultrasonic-sensor.html Adding the ultrasonic sensor 2014-06-29T11:37:00+00:00 GC <p>Hi everyone.</p> <p>In this post I’ll show the new circuit from the addition of the ultrasonic sensor, the code I uploaded to the Arduino to drive it, and a video to demonstrate its functioning. <!--more--></p> <div> <p> Here is the circuit. </p> <div style="clear: both; text-align: center;"> <a style="margin-left: 1em; margin-right: 1em;" href="http://giov.dev/images/IMG_20140626_194620.jpg"><img class="" src="http://giov.dev/images/IMG_20140626_194620.jpg" alt="" width="400" height="295" border="0" /></a> </div> <div style="clear: both; text-align: center;">   </div> <div style="clear: both; text-align: left;"> Nothing much going on here, the HC-SR04 needs 5V, so it's connected to the 5V pin, and obviously to ground. The Trig and Echo pins of the sensor are connected directly to two of the digital pins of the Arduino, D11 and D12. </div> <div style="clear: both; text-align: left;">   </div> <div style="clear: both; text-align: justify;"> <p> An ultrasonic sensor works like this: </p> <p> 1 &#8211; You send a short pulse to trigger the vibration of the transducer. Like ringing a bell.<br /> 2 &#8211; The sound waves produced by the transducer travel, hit something and are reflected, travelling back to the sensor, and making the transducer vibrate again. For this sensor there are two separate transducers, one for production, one for reception.<br /> 3 &#8211; The transducer, because of the vibration, produces a current, which is a function of the distance travelled by the sound waves.<br /> 4 &#8211; Read out that current and, knowing the function f(current)=distance, you have your distance! </p> <div style="clear: both; text-align: justify;">   </div> <div style="clear: both; text-align: justify;"> The sensor I&#8217;m using is quite smart, and instead of returning a continuous current to be read out and processed &#8211; the function f(current) = distance is easily nonlinear &#8211; it returns a digital signal, where the duration of the high value is linearly proportional to the distance of the sensed obstacle. Which makes things much easier. </div> <div style="clear: both; text-align: justify;">   </div> <div style="clear: both; text-align: justify;"> To make things even easier, there&#8217;s a <a href="https://code.google.com/p/arduino-new-ping/">nice ready-to-use ultrasonic sensor library for Arduino</a> which supports the HC-SR04. Cool! </div> <div style="clear: both; text-align: justify;">   </div> <div style="clear: both; text-align: justify;"> Here&#8217;s a video demonstration of the project at this point. </div> <div style="clear: both; text-align: justify;">   </div> <div style="clear: both; text-align: center;"> <p> </p> <p> &nbsp; </p> </div> <div style="clear: both; text-align: left;"> And here&#8217;s the code I used, adapted from the basic example of the New Ping library. </div> <figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="c1">// ------------------------------------------------------------------------ // This code reads the ultrasonic sensor about 20 times per second, and makes the motor vibrate // in inverse proportion to the distance (the less the distance, the more the vibration). // Also, to take out some noise, the last six readings are averaged. // The code is adapted from the New Ping library basic example. // ------------------------------------------------------------------------ </span> <span class="cp">#include </span> <span class="cp">#define TRIGGER_PIN 12 // Arduino pin tied to trigger pin on the ultrasonic sensor. #define ECHO_PIN 11 // Arduino pin tied to echo pin on the ultrasonic sensor. #define MAX_DISTANCE 200 // Maximum distance we want to ping for (in centimeters). Maximum sensor distance is rated at 400-500cm. </span> <span class="cp">#define MOTOR_PIN 3 // Arduino pin tied to motor control. </span> <span class="cp">#define N_READINGS 6 // Number of readings to average to get rid of some noise. </span> <span class="n">NewPing</span> <span class="n">sonar</span><span class="p">(</span><span class="n">TRIGGER_PIN</span><span class="p">,</span> <span class="n">ECHO_PIN</span><span class="p">,</span> <span class="n">MAX_DISTANCE</span><span class="p">);</span> <span class="c1">// NewPing setup of pins and maximum distance. </span> <span class="kt">int</span> <span class="n">read_array</span><span class="p">[</span><span class="n">N_READINGS</span><span class="p">];</span> <span class="c1">//Array that stores the last N_READINGS to average </span> <span class="kt">int</span> <span class="n">read_count</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="kt">float</span> <span class="n">ave_distance</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="kt">void</span> <span class="nf">setup</span><span class="p">()</span> <span class="p">{</span> <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o">&amp;</span><span class="n">lt</span><span class="p">;</span><span class="n">N_READINGS</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">){</span> <span class="c1">// Initialize readings to 0 </span> <span class="n">read_array</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="p">}</span> <span class="p">}</span> <span class="kt">void</span> <span class="nf">loop</span><span class="p">()</span> <span class="p">{</span> <span class="k">if</span><span class="p">(</span><span class="n">read_count</span><span class="o">&amp;</span><span class="n">gt</span><span class="p">;</span><span class="n">N_READINGS</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="n">read_count</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">delay</span><span class="p">(</span><span class="mi">50</span><span class="p">);</span> <span class="c1">// Wait 50ms between pings (about 20 pings/sec). 29ms should be the shortest delay between pings. </span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">uS</span> <span class="o">=</span> <span class="n">sonar</span><span class="p">.</span><span class="n">ping</span><span class="p">();</span> <span class="c1">// Send ping, get ping time in microseconds (uS). </span> <span class="kt">int</span> <span class="n">distance</span> <span class="o">=</span> <span class="n">uS</span> <span class="o">/</span> <span class="n">US_ROUNDTRIP_CM</span><span class="p">;</span> <span class="c1">// Compute distance from known constant (defined in New Ping library). </span> <span class="n">read_array</span><span class="p">[</span><span class="n">read_count</span><span class="p">]</span> <span class="o">=</span> <span class="n">distance</span><span class="p">;</span> <span class="c1">// Populate readings array. </span> <span class="kt">int</span> <span class="n">sum</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="c1">// needed for average </span> <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o">&amp;</span><span class="n">lt</span><span class="p">;</span><span class="n">N_READINGS</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">){</span> <span class="n">sum</span> <span class="o">+=</span> <span class="n">read_array</span><span class="p">[</span><span class="n">i</span><span class="p">];</span> <span class="p">}</span> <span class="n">ave_distance</span> <span class="o">=</span> <span class="n">sum</span><span class="o">/</span><span class="kt">float</span><span class="p">(</span><span class="n">N_READINGS</span><span class="p">);</span> <span class="c1">//average of last n readings </span> <span class="k">if</span> <span class="p">(</span><span class="n">ave_distance</span> <span class="o">&amp;</span><span class="n">lt</span><span class="p">;</span> <span class="mi">100</span> <span class="o">&amp;&amp;</span> <span class="n">distance</span><span class="o">!=</span><span class="mi">0</span><span class="p">){</span> <span class="c1">// 0 corresponds also to distance over sensor limit </span> <span class="n">analogWrite</span><span class="p">(</span><span class="n">MOTOR_PIN</span><span class="p">,</span> <span class="mi">255</span><span class="o">-</span><span class="kt">int</span><span class="p">(</span><span class="n">ave_distance</span><span class="o">*</span><span class="mi">1</span><span class="p">.</span><span class="mi">6</span><span class="p">));</span> <span class="c1">//1.6 is found empirically, by experimenting </span> <span class="p">}</span> <span class="k">else</span><span class="p">{</span> <span class="n">analogWrite</span><span class="p">(</span><span class="n">MOTOR_PIN</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span> <span class="c1">// needed, otherwise the pin will keep sending the last value sent </span> <span class="p">}</span> <span class="n">read_count</span><span class="o">++</span><span class="p">;</span> <span class="p">}</span></code></pre></figure> <div style="clear: both; text-align: justify;"> The code is pretty straightforward, there isn&#8217;t much to explain I guess. The 1.6 value I used to adjust the amount of vibration of the motor was obtained experimentally, by trial and error. It is high enough so that the motor still vibrates when the distance is at the maximum. If a lower value were to be used, then the motor wouldn&#8217;t receive enough current to start vibrating at distances near the limit.I&#8217;m starting to think that maybe having a vibration motor is not that great an idea. Its expressive range is very limited, as you can listen from the video, the vibration doesn&#8217;t seem to change much from the nearest distance to the farthest. Also, it consumes a great deal power, which I could use to add more sensors to the glasses.<br /> To explore other options, I bought two piezoelectric transducers, which should need very very little power, and much more control over their vibration is possible.For the next post, I&#8217;ll experiment with the piezoelectric tranducers. I think it won&#8217;t be a circuit post, as a piezoelectric transducer only needs a resistor, and nothing more, as far as I understand. Instead, given that there&#8217;s more things you can do with a piezoelectric transducer, like controlling amplitude and frequency of vibration, more attention will be given to the code.I&#8217;ll keep you posted! </div> </div> </div> 2014-06-29T11:37:00+00:00