Posts about clihttp://cestlaz.github.io/categories/cli.atom2018-09-19T23:47:52ZMike ZamanskyNikolaUser friendly - I do not think that means what you think it meanshttp://cestlaz.github.io/posts/user-friendly/2016-09-28T11:13:37-04:002016-09-28T11:13:37-04:00Mike Zamansky<div id="outline-container-org4354684" class="outline-2">
<h2 id="org4354684"></h2>
<div class="outline-text-2" id="text-org4354684">
<p>
Saw this post the other day: <a href="https://pupeno.com/2016/09/26/emacs-is-hurting-clojure/">Emacs is hurting Clojure</a> and this response: <a href="http://irreal.org/blog/?p=5597">Is Emacs Really HJard to Learn / Use</a>.
</p>
<p>
This called to mind those countless discussions about tools being user
friendly. I'm frequently on the unpopular side of these
discussions. Linux vs Windows vs Mac, Shell vs GUI, Emacs vs
fill-in-the-blank-editor-or-IDE and on and on.
</p>
<p>
Don't use Emacs, it isn't user
friendly. Don't use Linux, it isn't user friendly.
</p>
<p>
The thing is, I don't think that word means what they think it does.
</p>
<p>
I tell my kids that Linux and the shell is amazingly user friendly and
Windows and Macs aren't user friendly at all. Windows, rather, is
"Learner Friendly." For years, it's been easy to learn what you can do
on Windows or Mac (as used via the GUI, not dropping to a shell) but
the truth is you can't do that much. They're designed as program
loaders not tool kits.
</p>
<p>
You can learn how to get around and load programs and files very
quickly - much more quickly at first than using a command line
interface. The problem is, at that point, you hit a wall. No doubt
those operating systems are more learner friendly than Linux and the
shell but once you learn the command line, it can be leveraged for all
sorts of things and pretty easily at that - user friendly (see these
posts for some examples: <a href="http://localhost:8000/categories/cli/">Shell posts</a>).
</p>
<p>
This is important because we have our students use all sorts of tools
and we should be thinking about things like usability but we really
should be thinking about them a little differently.
</p>
<p>
First, is it learner friendly. Part of this is cost of
entry. Linux has a high cost of entry. On the other hand, while Emacs
might look uglier than other modern editors, the cost of entry is
pretty much the same as any other editor - click or type a command and then you can use
the mouse and drop down menus to get started. Vim, in my opinion, on
the other hand has a higher cost of entry due to its modal nature.
</p>
<p>
So, there's no reason not to use Emacs (vs another editor) but you
might pause before using Linux. My approach is to start as if it were
Windows - use the gui, icons and menus, and then slowly introduce the
command line.
</p>
<p>
Once your past the cost of entry, the we can think about being learner
friendly. As I mentioned, Linux, not so much. Emacs, sure - just like
any other editor with drop downs that then give you keyboard
equivalents.
</p>
<p>
The keyboard equivalents lead us to another part of learner friendly
– discoverability - can you easily discover new things about the
tool. Here Emacs shines. Linux, maybe not so much.
</p>
<p>
Once past learner friendly we get to user friendly. The core question
is "can I do what I need to and want to do easily." For most people, it
comes down to - the tool I know is user friendly and the one I don't
know and don't want to take the time to know isn't. There are some
things that are cumbersome no matter what, but really, when we say
user friendly, most people mean "what I like." To me, it's a little
more - is the tool expressive and powerful as well.
</p>
<p>
So, why is this important? First, we shouldn't allow our biases to
color the way we expose our kids to tools but at the same time, we
should pay attention to learnability, cost of entry, discoverability,
and indeed, true user friendliness.
</p>
<table border="2" cellspacing="0" cellpadding="6" rules="all" frame="border" border-spacing="10px" padding="10px">
<colgroup>
<col class="org-left">
<col class="org-left">
<col class="org-left">
</colgroup>
<thead>
<tr>
<th scope="col" class="org-left"> </th>
<th scope="col" class="org-left">Low Power</th>
<th scope="col" class="org-left">High Power</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left"><b><b>Low Learnability</b></b></td>
<td class="org-left">CP/M</td>
<td class="org-left">Linux Shell, Mac Terminal</td>
</tr>
<tr>
<td class="org-left"><b><b>High Learnability</b></b></td>
<td class="org-left">Windows / Mac OS GUI</td>
<td class="org-left">Most editors, including Emacs</td>
</tr>
</tbody>
</table>
<p>
So, let's stop with this "your tool isn't easy to use" nonsense. Any
tool we don't know isn't easy to use until we learn it. Let's focus on
the path to learning the useful ones.
</p>
</div>
</div>Shell short - tagging old posts in Nikolahttp://cestlaz.github.io/posts/shell-short-tagging/2016-05-15T09:58:44-04:002016-05-15T09:58:44-04:00Mike Zamansky<div id="outline-container-orgf31ea35" class="outline-2">
<h2 id="orgf31ea35"></h2>
<div class="outline-text-2" id="text-orgf31ea35">
<p>
Quick post to add to the recent command line fu I've been writing
about.
</p>
<p>
<a href="https://twitter.com/dougpete">Douglas Peterson</a> had another <b><b>Whatever happened to</b></b> post. This time
on <a href="https://dougpete.wordpress.com/2016/05/15/whatever-happened-to-6/">Logo</a>. I wanted to reply, talk about NetLogo and link to some of my
old NetLogo posts to help show how cool it is.
</p>
<p>
Nikola supports tags, makes a nice <a href="http://cestlaz.github.io/categories">tags</a> page and for each tag, a nice
page of all the <a href="http://cestlaz.github.io/categories/netlogo">links</a>.
</p>
<p>
Nikola has a plugin <b><b>tags</b></b> which lets you manage tags from the
command line. For instance:
</p>
<div class="highlight"><pre><span></span>nikola tags -a netlogo posts/somepost.org
</pre></div>
<p>
Would add the tag <b><b>netlogo</b></b> to the specified post.
</p>
<p>
The problem: The tags plugin only works if the post has a <b><b>tag:</b></b>
line already present in it's header comment and I hadn't put them in
my older posts.
</p>
<p>
I had a bunch of posts, all of them in one directory. All the new ones
were <b><b>.org</b></b> files and had the tag slug. The others were <b><b>.md</b></b> markdown
files and <b><b>.html</b></b> html files.
</p>
<p>
Here's what a typical top block looks like:
</p>
<div class="highlight"><pre><span></span><!--
.. title: Looking for interesting questions
.. slug: 2010-01-03-looking-for-interesting-questions.html
.. date: 2010-01-03
.. type: text
-->
</pre></div>
<p>
Sed to the rescue. Here's the what I ended up typing (from within the
posts directory) to add the
<b><b>tags</b></b> slug to the top comments right above the <b><b>.. type: text:</b></b> line:
</p>
<div class="highlight"><pre><span></span>ls *md *html | while read filename
do
sed "/type: text/ i .. tags: " $filename
done
</pre></div>
<p>
A line at a time:
</p>
</div>
<div id="outline-container-org3f82fe8" class="outline-3">
<h3 id="org3f82fe8">ls *md *html</h3>
<div class="outline-text-3" id="text-org3f82fe8">
<p>
This lists all the files with that end in <b><b>md</b></b> or <b><b>html</b></b>
</p>
</div>
</div>
<div id="outline-container-org3ecda0a" class="outline-3">
<h3 id="org3ecda0a">| while read filename</h3>
<div class="outline-text-3" id="text-org3ecda0a">
<p>
The vertical bar (pipe) sends the output of <b><b>ls</b></b> into the <b><b>while read</b></b>
command. The <b><b>while</b></b> command sets up a loop which, each time
through, reads the next input and places into the variable
<b><b>filename</b></b>. The body of the loop is between the <b><b>do</b></b> and the
<b><b>done</b></b>.
</p>
</div>
</div>
<div id="outline-container-orgc417ef1" class="outline-3">
<h3 id="orgc417ef1">sed "/ type: text/ i .. tags: " $filename</h3>
<div class="outline-text-3" id="text-orgc417ef1">
<p>
Sed is the stream editor. The stuff between the slashes finds the line
with the text <b><b>type: text</b></b> in it. The <b><b>i</b></b> inserts a line above and
the rest of the stuff in the quotes is what to insert. The
<b><b>$filename</b></b> expands to each filename, one each time through the
loop.
</p>
</div>
</div>
</div>
<div id="outline-container-org50c42d2" class="outline-2">
<h2 id="org50c42d2"></h2>
<div class="outline-text-2" id="text-org50c42d2">
<p>
Now all of my files have blank <b><b>tag</b></b> slugs so I can find my netlogo
posts and tag them:
</p>
<div class="highlight"><pre><span></span>nikola tags -a netlogo `grep -i -l netlogo posts/*`
</pre></div>
<p>
Any command in backticks expands to the result of the commmand. The
grep command has two argiments: <b><b>-i</b></b> means ignore case so it will
find netlogo, NetLogo, NETLOGO, etc.. The <b><b>-l</b></b> tells grep to just
output the filenames. So, the grep command will expand to a list of
files that mention netlogo. The full command adds the netlogo tag to
all of them.
</p>
<p>
So, just a bit of quick shell scripting and I've:
</p>
<ul class="org-ul">
<li>modified all old posts to accept tags.</li>
<li>added the netlogo tag to all my netlogo posts.</li>
</ul>
<p>
You can find all those posts <a href="http://cestlaz.github.io/categories/netlogo">here</a>.
</p>
</div>
</div>REPOST - Shell games - who confirmed attendancehttp://cestlaz.github.io/posts/duplicate-emails/2016-05-12T09:52:25-04:002016-05-12T09:52:25-04:00Mike Zamansky<div id="outline-container-orgf3a7807" class="outline-2">
<h2 id="orgf3a7807">Repost</h2>
<div class="outline-text-2" id="text-orgf3a7807">
<p>
This is a repost from March 2015. It didn't transfer when I rebooted the blog.
</p>
</div>
</div>
<div id="outline-container-orgdda01a4" class="outline-2">
<h2 id="orgdda01a4">Original</h2>
<div class="outline-text-2" id="text-orgdda01a4">
<p>
Quick post on why I love the Unix command line.
</p>
<p>
We're busy organizing <a href="http://cstuy.org">CSTUY's</a> first <a href="http://cstuy.org/programs/defhacks">hackathon</a>. It's going to be at
<a href="http://sumall.com">SumAll</a>, where we hold our weekly <a href="http://cstuy.org/programs/hacking_sessions">hacking sessions</a> but while taking
registration, we had a little program.
</p>
<p>
The kids signed up on a Google doc but we all know the story – when
people sign up for a free event, even one with free food and t-shirts,
many don't show. I asked all of the applicants to confirm by filling
out a second Google doc.
</p>
<p>
Then it got to reminder time - I wanted to send an email out to all
those kids who signed up on the first form, but hadn't confirmed on
the second.
</p>
<p>
Two Google spreadsheets with an email field. I needed all the people
on sheet 1 that weren't on sheet 2. I'm sure there's some
spreadsheet-fu that accomplishes this, but nothing I know. I also
could have written a little python script which isn't so bad, but this
was a perfect time to turn to the shell.
</p>
<p>
So, here's how a command line guy would do this.
</p>
<p>
To start, I put the emails in two files: e1 and e2. The first has
all the original applicants, the second those that confirmed.
</p>
<table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<colgroup>
<col class="org-left">
<col class="org-left">
<col class="org-left">
</colgroup>
<thead>
<tr>
<th scope="col" class="org-left">e1</th>
<th scope="col" class="org-left"> </th>
<th scope="col" class="org-left">e2</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left">a@a.com</td>
<td class="org-left"> </td>
<td class="org-left">b@b.com</td>
</tr>
<tr>
<td class="org-left">b@b.com</td>
<td class="org-left"> </td>
<td class="org-left">F@f.com</td>
</tr>
<tr>
<td class="org-left">c@c.com</td>
<td class="org-left"> </td>
<td class="org-left">c@c.com</td>
</tr>
<tr>
<td class="org-left">d@d.com</td>
<td class="org-left"> </td>
<td class="org-left">d@d.com</td>
</tr>
<tr>
<td class="org-left">e@e.com</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
<tr>
<td class="org-left">f@f.com</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
<tr>
<td class="org-left">g@g.com</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
<tr>
<td class="org-left">h@h.com</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
</tbody>
</table>
<p>
If we put these lists together, any email that appears twice would
indicate that it's the email of someone that confirmed entry. Here we
use cat to catenate e1 and e2 and pipe them through sort.
</p>
<div class="highlight"><pre><span></span>cat e1 e2 | sort
</pre></div>
<p>
First problem –the upper case F – let's use <b><b>tr</b></b> to make
everything lower case:
</p>
<div class="highlight"><pre><span></span>cat e1 e2 | tr A-Z a-z | sort
</pre></div>
<p>
Now we can see the duplicates next to each other. Next, <b><b>uniq
-c</b></b> tells us how many times each line appears:
</p>
<div class="highlight"><pre><span></span>cat e1 e2 | tr A-Z a-z | sort | uniq -c | sort
</pre></div>
<p>
I added the sort at the end, but we didn't need it.
</p>
<p>
Here's what we get:
</p>
<p class="verse">
1 a@a.com<br>
1 c@c.com<br>
1 c@c.dom<br>
1 e@e.com<br>
1 g@g.com<br>
1 h@hc.om<br>
2 b@b.com<br>
2 d@d.com<br>
2 f@f.com<br>
</p>
<p>
To pull out the ones that haven't replied I used egrep with a regex
that means "any line that starts with 1 or more spaces followed by the
number 1":
</p>
<div class="highlight"><pre><span></span>cat e1 e2 | tr A-z a-z | sort | uniq -c | egrep "^ +1"
</pre></div>
<p>
and finally to isolate the emails using sed which removes the spaces
and number 1 from the beginning of the line:
</p>
<div class="highlight"><pre><span></span>cat e1 e2 | tr A-z a-z | sort | uniq -c | egrep "^ +1" | sed "s/\ \+1 //g"
</pre></div>
<p>
Each of the little utilities aren't all too useful by themselves but
if you learn them over time you start thinking about how you can
combine them to solve problems.
</p>
<p>
If you think this way and know some basic tools, all of a sudden all
manner of text manipulation problems become pretty easy.
</p>
</div>
</div>BASH scripting?http://cestlaz.github.io/posts/bash-scripting/2016-05-12T08:39:25-04:002016-05-12T08:39:25-04:00Mike Zamansky<div id="outline-container-orgc8b556c" class="outline-2">
<h2 id="orgc8b556c"></h2>
<div class="outline-text-2" id="text-orgc8b556c">
<p>
Over in the Facebook AP Computer Science Teachers group someone asked
for thoughts on covering BASH scripting as a post AP topic.
</p>
<p>
A number of us made suggestions. I linked to <a href="http://cestlaz.github.io/posts/2014-07-06-spreadsheet-vs-cli.html/#.VzR7Et9vETt">this</a> old blog post.
</p>
<p>
One group member said she asked around for similar suggestions and the
response she got was "vi and awk." I wanted to jokingly respond "and
after they suggested that they got into their time machine and went
back to the 70's."
</p>
<p>
In all seriousness though, I think that suggesting specific tools or
commands is off base.
</p>
<p>
The important thing to know about Vi is how to get out of it but it
isn't really a tool in the scripting sense. I do think students should
spend a good amount of time learning a powerful editor and should try
bot Emacs (my choice) and Vim but that's another story.
</p>
<p>
I also use AWK but as it's a programming language in it's own right,
I'm not sure if I'd introduce it right off the bat.
</p>
<p>
There are a number of important ideas kids can take away from learning
some Linux (or other Unix flavor):
</p>
<ul class="org-ul">
<li>There's something out there besides Windows and MacOS</li>
<li>All about free software</li>
<li><a href="https://en.wikipedia.org/wiki/Unix_philosophy">The Unix Philosophy</a></li>
</ul>
<p>
That last one is the biggie and more specifically, there's a huge
upside in teaching kids the value of "OS as Toolset" where they can
compose the many tools that comprise the Linux experience to get
things done.
</p>
<p>
I gave an example of that in the <a href="http://cestlaz.github.io/posts/2014-07-06-spreadsheet-vs-cli.html/#.VzSBHt9vETu">post</a> I previously linked to.
</p>
<p>
For the teacher, that means wrapping your head around that way of
working. Living in the shell and using pipes to connect program to
progarm to program.
</p>
<p>
I'd recommend getting into a time
machine ourselves and taking a look at:
</p>
<div class="figure">
<p><a href="https://en.wikipedia.org/wiki/The_Unix_Programming_Environment" width="250px" align="center"><img src="http://upload.wikimedia.org/wikipedia/en/4/43/English4.gif" alt="English4.gif" width="250px" align="center"></a>
</p>
</div>
<p>
It's dated but it's really a great book on getting into the Unix way
of doing things, particularly the chapter about filters. It also has
one of the best and clearest introductions to writing a compiler in
the chapter on program development.
</p>
<p>
As I said, it is dated - shells are much easier to use and much more
robust, there are many more tools now, and they've evolved but it's
really a must read book.
</p>
<p>
In terms of tools, I get a lot of mileage out of:
</p>
<table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<colgroup>
<col class="org-left">
<col class="org-left">
<col class="org-left">
<col class="org-left">
</colgroup>
<thead>
<tr>
<th scope="col" class="org-left">command</th>
<th scope="col" class="org-left">description</th>
<th scope="col" class="org-left">example</th>
<th scope="col" class="org-left">explanation</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left">cat</td>
<td class="org-left">catenate or display a file</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
<tr>
<td class="org-left">tr</td>
<td class="org-left">Translate characters</td>
<td class="org-left">tr A-Z a-z</td>
<td class="org-left">convert upper to lower case</td>
</tr>
<tr>
<td class="org-left">sed</td>
<td class="org-left">Stream editor</td>
<td class="org-left">sed "s/a/b/g"</td>
<td class="org-left">Replace all a with b</td>
</tr>
<tr>
<td class="org-left">wc</td>
<td class="org-left">word count</td>
<td class="org-left"> </td>
<td class="org-left">counts words lines and chars</td>
</tr>
<tr>
<td class="org-left">cut</td>
<td class="org-left">cut columns</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
<tr>
<td class="org-left">sort</td>
<td class="org-left">sort lines</td>
<td class="org-left"> </td>
<td class="org-left"> </td>
</tr>
</tbody>
</table>
</div>
</div>
<div id="outline-container-org361c68b" class="outline-2">
<h2 id="org361c68b"></h2>
<div class="outline-text-2" id="text-org361c68b">
<p>
A nice simple thing you can do with these is clean data. Let's say you
want to do some analytics on a book from Project Gutenberg. You might
want to convert all non letters to spaces, and all letters to lower
case:
</p>
<div class="highlight"><pre><span></span>cat book.txt | sed "s/[^a-zA-Z ]/ /g | tr A-Z a-z"
</pre></div>
<p>
That sends book.txt into sed which uses a regular expression to convert
no space and letters to spaces. The tr command converts all upper case
letters to lower case.
</p>
<p>
If you want one word per line, add:
</p>
<div class="highlight"><pre><span></span>| sed "s/\n/g"
</pre></div>
<p>
and maybe get rid of blank lines:
</p>
<div class="highlight"><pre><span></span>| sed "/^$/d"
</pre></div>
<p>
We can now count the number of words in the file using <b><b>*wc</b></b> or even
get counts of all the words:
</p>
<div class="highlight"><pre><span></span>| sort | uniq -c | sort -n
</pre></div>
<p>
<b><b>sort</b></b> will sort all the lines, <b><b>uniq -c</b></b> will compress the lines that are
adjacent and the same and give you a count and then <b><b>sort -n</b></b> will
sort the results numerically.
</p>
<p>
I wrote another post a while ago about using the shell to detect
who responded on a Google form. It looks like it didn't convert when
I moved to my current blogging platform - I'll repost that shortly.
</p>
</div>
</div>Should We Teach HTML?http://cestlaz.github.io/posts/teaching-html/2016-05-09T19:08:58-04:002016-05-09T19:08:58-04:00Mike Zamansky<p>
Yesterday, <a href="https://twitter.com/dougpete">Doug Peterson</a> wrote a "Whatever happened to" post subtitled
<a href="https://dougpete.wordpress.com/2016/05/08/whatever-happened-to-5/">HTML as an essential 21st Century skill?</a> It's a nice post.
</p>
<p>
I left a comment but thought I'd elaborate here.
</p>
<p>
No, knowing HTML is not programming - it's markup. Even so, when I
help people design CS programs, I'll frequently recommend starting
with HTML or at least introducing it early.
</p>
<p>
Why?
</p>
<p>
It's a gateway and not just to programming.
</p>
<p>
HTML is pretty easy, you want <b><b>something</b></b> bold, you just wrap the
word in <code><b></code> and <code></b></code>:
</p>
<div class="highlight"><pre><span></span><b>something</b>
</pre></div>
<p>
It's also empowering and demystifies the web. Kids can create a simple
web page and load it right into their browser.
</p>
<p>
It's true that today's web pages are chock-full-o javascript and css
but with just the basics, students can get the idea. You can also show
them pages by right clicking and viewing source.
</p>
<p>
You can even have them change a live page.
</p>
<p>
Try it.
</p>
<p>
Right click on the top of this page where it says "Musings about…"
Chose <b><b>inspect element</b></b>. In the "debugger" window double click the
text, change it and hit enter. This is just temporary - just reload
the page but it's pretty neat for a kid to change an article and then
screenshot it.
</p>
<p>
HTML is also a nice stepping stone towards coding. You're working in a
plain text editor by adding special code words to basic text which are
then interpreted by, in this case, the web browser.
</p>
<p>
The big reason for teaching html actually goes beyond this. Next step
after learning HTML is having the kids programatically generating web
pages in whatever language you're using for the class. I like using
Python. This requires a little infrastructure setup to serve kids work
but then there are two huge wins.
</p>
<p>
First, as the kids learn programming, instead of just printing out
results, they can make a web site with their results and share it with
friends, family, and the world.
</p>
<p>
The other big bonus is that kids might be able to leverage take these
skills to other classes. If the student has a history paper to write,
maybe the teacher will accept a history web site where the student can
write code to do their analytics and build nice looking tables and
graphs with results.
</p>
<p>
So while knowledge of HTML in and of itself isn't really needed
anymore it's still an important part of the programs I build.
</p>Shell games - who confirmed attendencehttp://cestlaz.github.io/posts/2015-03-19-duplicate-emails.html/2015-03-19T00:00:00-04:002015-03-19T00:00:00-04:00Mike Zamansky<style>
div.center {text-align:center;}
</style>
<p>
Quick post on why I love the Unix command line.
</p>
<p>
We're busy organizing <a href="http://cstuy.org">CSTUY's</a> first <a href="http://cstuy.org/programs/defhacks">hackathon</a>. It's going to be at
<a href="http://sumall.com">SumAll</a>, where we hold our weekly <a href="http://cstuy.org/programs/hacking_sessions">hacking sessions</a> but while taking
registration, we had a little program.
</p>
<p>
The kids signed up on a Google doc but we all know the story – when
people sign up for a free event, even one with free food and t-shirts,
many don't show. I asked all of the applicants to confirm by filling
out a second Google doc.
</p>
<p>
Then it got to reminder time - I wanted to send an email out to all
those kids who signed up on the first form, but hadn't confirmed on
the second.
</p>
<p>
Two Google spreadsheets with an email field. I needed all the people
on sheet 1 that weren't on sheet 2. I'm sure there's some
spreadsheet-fu that accomplishes this, but nothing I know. I also
could have written a little python script which isn't so bad, but this
was a perfect time to turn to the shell.
</p>
<p>
So, here's how a command line guy would do this.
</p>
<p>
To start, I put the emails in two files: e1 and e2. The first has
all the original applicants, the second those that confirmed.
</p>
<div class="org-src-container">
<pre class="src src-org">|</pre></div>Spreadsheet? I'd rather use the command line.http://cestlaz.github.io/posts/2014-07-06-spreadsheet-vs-cli.html/2014-07-06T00:00:00-04:002014-07-06T00:00:00-04:00Mike Zamansky<style>
div.center {text-align:center;}
</style>
<p>
Spreadsheets are terrific - we've all used them. I particularly like
Google spreadsheets - I use them all the time to collect data, usually
from students.
</p>
<div id="outline-container-sec-1" class="outline-2">
<h2 id="sec-1"></h2>
<div class="outline-text-2" id="text-1">
<ul class="org-ul">
<li>Go to Google Drive
</li>
<li>Make a form
</li>
<li>Send the form out to the students
</li>
<li>Wait
</li>
</ul>
<p>
All the data gets dumped into a Google spreadsheet. The trouble is,
what to do with it once it's in the spreadsheet.
</p>
<p>
The other day, I wrote on a few basic stats for our upcoming SHIP
program. The data I reported on was all collected in a spreadsheet. I
also collected participant and parent emails in the spreadsheet.
</p>
<p>
So, here's the task, compute some simple numbers form the spreadsheet
and also extract and use the email addresses.
</p>
<p>
I'm sure one could use some fancy spreadsheet magic to get the job
done, but I'm a command line wonk – here's how I take care of tasks
like these.
</p>
<p>
First, I downloaded the spreadsheet as a csv (comma separated value)
file. Each line looked something like this:
</p>
<div class="org-src-container">
<pre class="src src-csv">last,first,email,address,gender,grade,school,...
</pre>
</div>
<p>
First question, how many applicants did we have:
</p>
<div class="org-src-container">
<pre class="src src-sh">cat cstuy.csv | wc -l
</pre>
</div>
<p>
Which gave:
</p>
<pre class="example">
47
</pre>
<p>
The |, or pipe means take the output of the first command and send it
to the next one. <b>Cat</b> just outputs the original file and <b>wc -l</b> counts all
the lines in the file.
</p>
<p>
Next, how many young ladies:
</p>
<div class="org-src-container">
<pre class="src src-sh">cat cstuy.csv | grep female | wc -l
</pre>
</div>
<p>
The results:
</p>
<pre class="example">
17
</pre>
<p>
How many schools? Well, that's a little trickier. Here, I use a few
extra commands:
</p>
</div>
</div>
<div id="outline-container-sec-2" class="outline-2">
<h2 id="sec-2"></h2>
<div class="outline-text-2" id="text-2">
<ul class="org-ul">
<li>cut - this will cut out one column from the csv file - in this case
the school column (the -d, says use a comma as delimiter and -f7 for
field 7).
</li>
<li>sort - takes the lines and sorts them.
</li>
<li>uniq - eliminates duplicate lines in a sorted file
</li>
</ul>
<p>
Putting it all together:
</p>
<div class="org-src-container">
<pre class="src src-sh">cat cstuy.csv | cut -d, -f7 | sort | uniq | wc -l
</pre>
</div>
<p>
Results:
</p>
<pre class="example">
30
</pre>
<p>
Thirty different schools.
</p>
<p>
Finally I needed the emails - here I wanted to be able to paste them
into Gmail's bcc field. I could have just used <b>cat</b> and <b>cut</b> and
then used the mouse, but instead:
</p>
<div class="org-src-container">
<pre class="src src-sh">cat cstuy.csv | cut -d, -f3 | xclip -sel clip
</pre>
</div>
<p>
Then I can just do a Ctrl-v in Gmail and I'm good to go.
</p>
<p>
The cool thing is that the tools here - cut, sort, uniq, grep - are
all general purpose tools that do simple text manipulations. Once you
know them and a few others, you can really quickly and efficiently do
all sorts of data processing without even thinking about it. I still
go to the spreadsheet for data collection ad also for when I need more
hardcore formulas but for day to day manipulations, I'll take the
command line.
</p>
</div>
</div>Shell Games - an introductionhttp://cestlaz.github.io/posts/2014-02-04-shellgames-1-intro.md/2014-02-04T00:00:00-05:002014-02-04T00:00:00-05:00Mike Zamansky<div><p>A few weeks ago, I noticed this Twitter conversation between <a href="http://twitter.com/alfredtwo">Alfred Thompson</a> and <a href="http://twitter.com/keinath">Steve Keinath</a></p>
<p></p>
<blockquote class="twitter-tweet">I'd love to see an Intro to Linux (way more than just install) as a 3-hour workshop at <a href="https://twitter.com/search?q=%23CSTA14&src=hash">#CSTA14</a> <a href="https://twitter.com/csteachersa">@csteachersa</a>— Steve Keinath (@keinath) <a href="https://twitter.com/keinath/statuses/400333558603997184">November 12, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<blockquote class="twitter-tweet"><a href="https://twitter.com/alfredtwo">@alfredtwo</a> <a href="https://twitter.com/csteachersa">@csteachersa</a> Right. I know very little & would love a "zero to hero" Linux workshop.— Steve Keinath (@keinath) <a href="https://twitter.com/keinath/statuses/400335634297937920">November 12, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p></p>
<p>I briefly considered proposing a session for the conference but it was just a day or two before the deadline, I don't know if I'm going to be able to attend the conference, and besides, who said anything I proposed would be accepted.</p>
<p></p>
<p>Still, I liked the idea - I've been an educator for 23 years, a Linux user for most of that time and an Unix user for longer. I'm a firm believer in operating system as toolkit and so I think I'll take Steve and Alfred's suggestion and try to put together a series of posts on using Linux from a CS educators point of view.
</p><p></p>
<p>So, before we begin - a little background.</p>
<p></p>
<p>I can proudly say that I've been Windows free since about 2000. That's when I decided to wipe the lat traces of Microsoft from my hard drives. Prior to that I just booted up MS-DOS or Windows to play games or to use a Excel or Word.
</p><p></p>
<p>Since the early days of Linux - back before Slackware, I dual booted. Before Linux, I dialed into public Unix systems such as <a href="http://www.panix.com">Panix</a> or <a href="http://en.wikipedia.org/wiki/The_Big_Electric_Cat">The Big Electric Cat</a>. At home, I tried to make MS-DOS as Unix like as I could. I ran the <a href="http://en.wikipedia.org/wiki/MKS_Toolkit">MKS Toolkti</a>, and used my own shell (a project every young programmer should attempt).
</p><p></p>
<p>Why am I posting this now? It's a new semester and I find myself, as usual, leveraging the Linux shell. It was time to set up a mailing list for the class.</p>
<p></p>
<p>I'm able to go to our school's data system and grab a tab delimited file that looks something like this:
</p><p></p>
<p></p><blockquote><pre>
Code Section Period Last First ID Official Advisor OSIS Email
grY22tBs 01 6 Hxk Blu GFy 9272 7rr gEs 274989649 zlu3lxk@QylKR.oqy
grY22tBs 01 6 HiQqvlRu Blku 9918 7PP YHZHm 200878353 zzl8@yu.oqy
grY22tBs 01 6 plxk ClSKv 9226 7II PHXrNY 274661826 olxkvl@QylKR.oqy
grY22tBs 01 6 pxKk BqVxFl 9026 7II PHXrNY 224608174 zo6461@lqR.oqy
grY22tBs 01 6 pqxuk NRK 9234 7dd gHAMmNd 270217219 uRKo90@QylKR.oqy
</pre> </blockquote>
<p></p>
<p>It's tab delimited but I scrambled the letters so as to not reveal any student info.
</p><p></p>
<p>Oh, how did I do that scrambling? Easy. First, I combined some basic utilities to make a random permutation of the upper and lower case letters and stored them in a shell variable. Don't worry, I'll explain these commands in upcoming posts:
</p><p></p>
<blockquote>
perm=`echo "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ | sed "s/\(.\)/\1\n/g" | sort -R | tr -t "\\n" ":" | sed "s/[^a-zA-Z0-9,@]//g"`
</blockquote>
<p></p>
<p>Then I used tr (translate) to exchange the real letters for the matching letter in the random permutation:
</p><p></p>
<blockquote>
cat students.tsv | tr a-zA-Z $perm > students.scrambled
</blockquote>
<p></p>
<p>So back to the real work. I needed to isolate the students email addresses. The process:
</p><p></p>
<p></p><ol>
<li>convert the tabs to commas</li>
<li>Pull out the students in my AP class (code MKX22X) from the list of all students</li>
<li>Pull out the 10th column</li>
<li>These are the emails, save them to a file</li>
<p></p>
<p>So, I typed:
</p><p></p>
<blockquote>
cat students.tsv | grep MKS22X | sed "s/\t/,/g" | cut -d, -f10 > emails
</blockquote>
<p></p>
<p>grep filters out lines that have MKS22X in them and sed replaces the tabs (\t) with commas and cut pulls out the email addresses. It's all stored in a file named emails.
</p><p></p>
<p>Now, I just have to import these into my maillist software (mailman).
</p><p></p>
<p></p><blockquote>
add_members -r emails myclasslist
</blockquote>
<p></p>
<p>So, that's it, easy peasy.
</p><p></p>
<p>I'll be away for most of this week at the Tapia conference and then I'll be playing catch up, but I'm hoping to do a series of posts talking about my Linux toolset and how I use it.
</p><p></p>
<p>I hope you all find it interesting and maybe even useful.
</p><p></p></ol></div>Why we scripthttp://cestlaz.github.io/posts/2013-01-24-why-we-script.md/2013-01-24T00:00:00-05:002013-01-24T00:00:00-05:00Mike Zamansky<div><p>I tell my students "the cool thing about what we do is that if we're not happy with the way something works, we've got a shot at fixing it."</p>
<p>That came up this morning so I thought I'd share.</p>
<p>I
recently <a href="http://cestlaz.github.com/2012/12/09/real-projects.html#.UQFhh1L6s7x">posted</a>
about the in-term projects my Software Development kids were working
on. Well, now it's time to grade their final projects.</p>
<p>The code is up on GitHub. This morning I was faced with independently
going to every project page and cloning each one:</p>
<div align="center">
<a href="http://cestlaz.github.com/img/github-projects.png" rel="lightbox">
<img width="50%" src="http://cestlaz.github.io/img/github-projects.png" class="" alt="">
</a>
</div>
<p>Not fun!!!!</p>
<p>There had to be a better way. Fortunately all the projects were under a single "organization" and a little digging into the GitHub API reminded me that I could use this url:</p>
<pre>
https://api.github.com/orgs/stuycs-ml7-projects/repos
</pre>
<p>which brought up all this nice JSON data.</p>
<div align="center">
<a href="http://cestlaz.github.io/img/github-api.png" rel="lightbox">
<img width="50%" src="http://cestlaz.github.io/img/github-api.png" class="" alt="">
</a>
</div>
<p>A little poking around in the data finds that each project url is part of a line that starts with "ssh_url." </p>
<p>a little wget, sed, grep and sh magic later:</p>
<pre>
urls=`wget --quiet -O - https://api.github.com/orgs/stuycs-ml7-projects/repos | grep ssh_url | sed "s/.*\(git.*\.git\).*/\1/g"`
for url in $urls
do
git clone git@$url
done
</pre>
<p>Now, as long as all the projects are under a single Github organization I can easily clone or pull them without having to navigate the Github web site. </p>
<p>Commandline FTW!!!!!!</p></div>