BULL SPEC

Against probably all better judgement, I've put a stake in the ground and hung a sign on it: BULL SPEC is open for submissions: Welcome to BULL SPEC. [More]

Lessons

As I ruminated on perhaps learning a lesson about which side projects are appropriate for a software engineer and father of two in a house without a soundproof music room, I realized that I have not properly learned one of the lessons I naïvely thought learned long ago, about getting work done to have in the end more -- and more quality -- time for non-work pursuits.

First, I think that I have come to believe that working on software projects as a side project from work (as a software engineer) are not right for me: (1) By the end of a day at work, particularly in the "crunch time" which has been shifting more and more to "all the time," often the last thing I want to do is sit back at a computer and write and test code, and in the other direction, after working all evening or weekend on my personal programming project my performance at work inevitably suffers; (2) it turns out that I am terrible at web design, and while this might be alleviated by study and practice, see #1, though as a personal development goal I hope to improve in this space through work-study at work; (3) potential IP conflicts with work.

Now, seeing this, what other side projects can I indulge in as an outlet for creative expression? The obvious ones to me are music and writing. Music has been hard to find time and workspace for, as having my equipment out is a disaster waiting to happen with two children under 3, and in any case playing guitar, trumpet, piano, synth, and singing after the kids go to bed is not an option. I'm still hoping to spend much of my December vacation recording new and old music and re-recording some even older songs, but it does not work out as a good fit.

But writing seems to fit. The kids demand stories at all times, so in a sense I'm "writing" all the time, even though half of those stories are "Marco plays soccer/baseball/hockey with his friends" or "Marco goes to the truck/tractor store with his friends." It's even a bedtime story which has uncorked the latest flurry of writing for me, "The Riddle of the Koi," even though yesterday I realized that more than likely that my Tolkien-reared subconscious had produced the "ROTK" initialization of the title. I can write without piles of gear and cords strewn about one or more rooms. I can write while waiting for a dentist appointment, or in an hour and a half of free time while the kids are blessedly napping at roughly the same time, or in the hour and a half after the kids go to bed. I can write! It is like rediscovering a vestigial limb, long forgotten.

Now, the naïveté. I have often thought to myself, proud of my supposed increasing wisdom, that if I had to go back and do college over again, I would actually study and work harder -- in order to in the result have more time for a personal life and personal pursuits, as well as getting the most out of the studies I was there to pursue in the first place. In college, far too often I put off work, put off studying, and in the end this caused me to spend more time and interrupt more personal choices to eventually make up that intellectual debt, and to do so in such a half-assed manner -- just enough to make the grade or pass the test -- that I missed even getting the real, life-long benefit of learning, despite the increased time cost.

But I realize that this is actually a lesson I have not really learned. At work, I too often let distractions sneak their way into my day, causing my work to suffer and slip, leading to work following me home as deadlines grow nearer and nearer, turning an 8 hour work day into 10-12, completely unnecessarily and completely foolishly and wastefully. On that note: to work! So that later, instead of paying the accrued interest of work debt, I can get to that topic which sparked the whole topic: writing the "ninja, pirates, and zombies set in feudal Japan" novelette -- and no, at least I hope not, it is not as trite as that sounds -- that wants its way out of my head: "A team of ninja sent to capture a pirate ship finds more than they bargained for when the corpses of the pirates lurch back to life, as one member of the ninja team's life comes full circle." [More]

Whiteboarding a terribly drawn cartoon

Well, more than two and a half years since the last post; appropriate the last one was about how being a 'new dad' changes one's perspectives on things.

OK, enough banter.

I finally think I am beginning to "grok" Twitter; I'd avoided it for a long while, and finally I see some fun in it: watching Neil Gaiman, William Gibson, and a few other folks tweet their way through their crazy days is actually fairly interesting, and through those tweets I've been introduced to a few new writers that I've come to like.

But then it gets weird.

I'd been following some good-natured cross-sniping between a few folks, followed a few of them, and, after a particularly interesting (to me) exchange, finally decided that I'd put marker to whiteboard and sketch some of it out in high drama. Very, very, very low art but it's a start; I don't think I've ever actually put a sketch online before, and these may be the first six sketches I've done in over a decade.

Illustration is certainly something I never developed. I like to think I did pretty well on the music side, and passably well on the writing side. But visual art has always escaped me. I doubt I will ever sketch anything worth looking at for more than a second of disgust and distaste, but a 50% improvement will be enough for me, and I actually found the "process" enjoyable. The "process?" Basically: sketch on an 8x10 whiteboard, snap a picture with my (7 year?) old Olympus digital, shrink it to 30% size (using MS Paint no less, if I keep going I'll write a script to do it) and upload to montsamu musings on drunkduck.com.

Unfortunately I have a dozen more drawings in mind for that story alone to reach its conclusion, and already there's another one in mind for the next chapter.
[More]

Finger-picking good

Whew. I haven't played the guitar this much since October. I played until my fingers hurt. Then I played a few more songs. [M]rs. montsamu and baby montsamu were the receptive audience, although both settled in for a nap by the end. I'm getting closer to putting closing the books on a new song called The Sparrow -- a departure from my usual kind of song in that it actually has an expressed meaning, and is intended for children. But I guess that is just one example as to how a new dad's life changes.

Sometimes a sparrow's ignorance
can be confused for divine providence
because sometimes
knowing how far can keep you from taking flight at all.

[More]

Erlang: parallel-map and parallel-foreach

In a post to erlang-questions last summer, Erlang on the niagara, Erlang super-hero Joe Armstrong posts a simple "parallel-map" implementation that he uses in place of some of his simple "iterative-foreach" calls to quickly achieve dramatic speed improvements by taking advantage of multiple processors. The post got me thinking, particularly in light of recent discussions on side-effects and purity, about the real differences and guarantees of such simple things as map and foreach.

Principally, map is (1) supposed to be called with a function with no side effects, (2) guarantees order of return to match the input list, and (3) makes no guarantee on order of function calls, while foreach is (1) used explicitly (and solely) for its side effects as (2) it has no return value (other than the atom ok for success) and (3) guarantees the order of function calls to match the order of the input list. Now, Joe's code was for a parallel-map implementation -- returning a list of results from the application of the given function, no guaranteed order of execution -- when it was actually being used to replace a foreach statement. It happened to work out nicely that his code did not rely on any order of execution promises.

But what if it did? And, anyway, it also made me wonder what cost was entailed with gathering the result list of a (potentially) large number of function calls, when in the end the result needed was only the lack of an error, and the eventual return of control to signify that the operation was complete?

But let's go back. First, let's look at how the standard Erlang lists module implements these two functions:

1> lists:map(fun(I) -> io:format("~w~n",[I]), I+1 end, lists:seq(1,10)).
1
2
3
4
5
6
7
8
9
10
[2,3,4,5,6,7,8,9,10,11]
2> lists:foreach(fun(I) -> io:format("~w~n",[I]) end, lists:seq(1,10)).
1
2
3
4
5
6
7
8
9
10
ok

Exactly as we expected. It just so happened that map made the function calls in the same order as the input list, but this wasn't guaranteed. What do I mean? A glance at the relevant documentation has the following to say about foreach:

This function is used for its side effects and the evaluation order is defined to be the same as the order of the elements in the list.

Contrast this with what it has to say about map:

This function is used to obtain the return values. The evaluation order is implementation dependent.

Fair enough. I put together a little first pass at a plists module (for, obviously, "parallel lists"):

-module(plists).
-export([pmap/2,pforeach/2,npforeach/2]).

pmap(F, L) ->
    S = self(),
    Pids = lists:map(fun(I) -> spawn(fun() -> pmap_f(S, F, I) end) end, L),
    pmap_gather(Pids).

pmap_gather([H|T]) ->
    receive
        {H, Ret} -> [Ret|pmap_gather(T)]
    end;
pmap_gather([]) ->
    [].

pmap_f(Parent, F, I) ->
    Parent ! {self(), (catch F(I))}.

pforeach(F, L) ->
  S = self(),
  Pids = pmap(fun(I) -> spawn(fun() -> pforeach_f(S,F,I) end) end, L),
  pforeach_wait(Pids).

pforeach_wait([H|T]) ->
  receive
    H -> pforeach_wait(T)
  end;
pforeach_wait([]) -> ok.

pforeach_f(Parent, F, I) ->
  _ = (catch F(I)),
  Parent ! self().

npforeach(F, L) ->
  S = self(),
  Pid = spawn(fun() -> npforeach_0(S,F,L) end),
  receive Pid -> ok end.

npforeach_0(Parent,F,L) ->
  S = self(),
  Pids = pmap(fun(I) -> spawn(fun() -> npforeach_f(S,F,I) end) end, L),
  npforeach_wait(S,length(Pids)),
  Parent ! S.

npforeach_wait(_S,0) -> ok;
npforeach_wait(S,N) ->
  receive
    S -> npforeach_wait(S,N-1)
  end.

npforeach_f(Parent, F, I) ->
  _ = (catch F(I)),
  Parent ! Parent.

Some things to notice here are that (1) my pmap is effectively the same as Joe's, but that (2) I've provided a pforeach here that works slightly differently in that it disregards return values instead of gathering them into a list. (Note, however, that like pmap it also disregards order of execution promise and so could not be a drop-in replacement for the iterative foreach.) For now, ignore the npforeach as I'll talk about it later.

Fine, anyway, let's play:

$ erl -smp +S 16 +A 2

Here I start Erlang, enabling the use of multiple processors, setting the internal scheduler to use 16 scheduler threads, and using 2 asynchronous threads.

Erlang (BEAM) emulator version 5.5.2 [source] [smp:16] [async-threads:2] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> c(plists).
{ok,plists}
2> plists:pforeach(fun(I) -> io:format("~w~n", [I]) end, lists:seq(1,10)).
2
3
4
5
6
7
1
8
9
10
ok
3> plists:pforeach(fun(I) -> io:format("~w~n", [I]) end, lists:seq(1,10)).
1
2
3
4
5
7
9
6
10
8
ok

OK! Here we go, we see that, as expected, the order of execution is certainly interleaved -- and not predictably. Let's move on to pmap:

4> plists:pmap(fun(I) -> io:format("~w~n",[I]), I + 1 end, lists:seq(1,20)).
1
3
4
5
6
7
8
10
9
11
12
13
14
15
16
17
18
19
20
2
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21]

Here we see that pmap also doesn't have any guaranteed order of execution (which it shouldn't) and that the return values are in proper order (which it should).

Now. Places to improve? Well, for one we could get all complicated-like and have options for a certain number of worker processes instead of "blindly" spawning a process for each element of a (potentially large!) input list. Another idea is spawning a "spawner" process to actually spawn the worker processes, so that our gathering can begin immediately without waiting for all the worker spawning to end. Yet another is, since we do not need any ordered return value from pforeach find a marginally better way of acknowledging process completion as they occur instead of always waiting for each process in order. (Now you can go back and look at npforeach if you like.) But mostly at that point you're just shuffling deck chairs. Actually just the simple difference between pmap and pforeach is mostly shuffling deck chairs. Certainly it "seems" like pforeach should be faster than pmap -- but lacking a Niagara of my own, mostly I end up testing the limits of my limited CPU and memory slice of my shared VPS server.

But, if anyone does happen to have an interesting amount of hardware lying around:

-module(plists_test).
-export([start/0]).

start() ->
  timer:start(),
  F = fun(I) -> math:pow(I,I) end,
  {T0,L} = timer:tc(lists,seq,[1,1000]),
  io:format("seq took ~w microseconds~n",[T0]),
  {T1,_V2} = timer:tc(plists,pmap,[F,L]),
  io:format("pmap took ~w microseconds~n",[T1]),
  {T2,_V1} = timer:tc(plists,pforeach,[F,L]),
  io:format("pforeach took ~w microseconds~n",[T2]),
  {T3,_V1} = timer:tc(plists,npforeach,[F,L]),
  io:format("npforeach took ~w microseconds~n",[T3]),
  ok.

And give it a twirl:

$ erl -smp +S 16 +A 2 -s plists_test -run init stop -noshell
seq took 109 microseconds
pmap took 90465 microseconds
pforeach took 104702 microseconds
npforeach took 112472 microseconds

As the input list gets larger, that "queue of processes instead of trying to spawn a process for each element of the list blindly" idea starts to look a whole lot better -- what good is spawning 10000 CPU-bound processes when you only a fraction of that in available CPU cores? [More]