00:02:18  * ghostbar_joined
00:05:24  * dapquit (Quit: Leaving.)
00:06:19  * ghostbarquit (Ping timeout: 276 seconds)
00:10:54  * fredkquit (Quit: Leaving.)
00:12:26  * yunongquit (Quit: Leaving.)
00:12:33  * notmattquit (Remote host closed the connection)
00:17:44  * jcspquit (Ping timeout: 252 seconds)
00:32:45  * bcantrillquit (Quit: Leaving.)
00:33:16  * bcantrilljoined
00:34:52  * bcantrillquit (Client Quit)
00:47:53  * AvianFluquit (Remote host closed the connection)
00:48:31  * AvianFlujoined
00:55:04  * mcavagejoined
01:03:08  * mcavagequit (Remote host closed the connection)
01:03:29  * notmattjoined
01:16:02  * notmattquit (Remote host closed the connection)
02:00:32  * bcantrilljoined
02:13:35  * mcavagejoined
02:21:31  * mcavagequit (Ping timeout: 276 seconds)
02:26:38  * notmattjoined
02:31:16  * notmattquit (Ping timeout: 276 seconds)
04:03:11  * bcantrillquit (Ping timeout: 260 seconds)
04:15:54  * chorrelljoined
04:16:10  * chorrellquit (Client Quit)
04:28:47  * AvianFlu_joined
04:32:10  * AvianFluquit (Ping timeout: 276 seconds)
04:46:05  * bixuquit (Remote host closed the connection)
04:54:01  * notmattjoined
05:31:23  * AvianFlu_quit (Remote host closed the connection)
06:19:15  * bcantrilljoined
06:22:49  * bcantrill1joined
06:24:26  <jzu_>bcantrill: Hi, a quick out-of-topic question: happen to know - is it normal that if I do pkg install zsh in Solaris 11 non-global Zone the package is also marked as installed in the Global Zone?
06:24:50  <jzu_>bcantrill: that kinda of fights against the ideology of Zones, but I guess it's unavoidable since basedir is /usr
06:25:03  <jzu_>s/kinda/kind/
06:26:25  * bcantrillquit (Ping timeout: 246 seconds)
06:29:07  <rmustacc>jzu_: I can assure you that he can't answer questions about S11.
06:29:15  <jzu_>hehe
06:29:25  <jzu_>think I'll try Googling!
06:29:28  <rmustacc>Especially when it comes to ips.
06:29:42  <jzu_>but anyways, that seems weird behavior, but on the other hand it's understandable
06:29:53  <jzu_>must be just a way IPS works
06:29:57  <rmustacc>But I would imagine someone in #illumos or one of the ips distros like oi/omnios would know much better.
06:30:05  <jzu_>yeah, gotta try that
06:30:19  <rmustacc>Given that we don't use ips at Joyent, we're one of the last folks to ask. ;)
06:30:48  <jzu_>ah, yeah correct =)
07:40:40  * bcantrill1quit (Quit: Leaving.)
07:47:05  * almostobsoletejoined
08:01:34  * mamashjoined
08:17:05  * almostobsoletequit (Ping timeout: 248 seconds)
08:33:09  * jcspjoined
09:05:50  * jcspquit (Ping timeout: 240 seconds)
09:39:44  * bixujoined
09:39:54  * bixuquit (Remote host closed the connection)
09:40:42  * bixujoined
09:57:43  * marsell_joined
10:00:44  * marsellquit (Ping timeout: 268 seconds)
10:00:44  * marsell_changed nick to marsell
11:15:26  * almostobsoletejoined
11:18:40  <almostobsolete>What's the best way to do logging from scripts on Manta?
11:41:53  * AvianFlujoined
11:47:58  * cburroughsjoined
12:16:07  * notmattquit (Remote host closed the connection)
12:31:03  * bixuquit (Remote host closed the connection)
12:46:37  * notmattjoined
12:49:23  <almostobsolete>Where can I see how much I've spent so far? Want to make sure it's not too massively expensive
12:54:49  * notmattquit (Ping timeout: 240 seconds)
12:58:11  <jperkin>almostobsolete: http://apidocs.joyent.com/manta/reports.html
12:59:18  <jperkin>as for logging from scripts, you could mput output back into manta from the job
13:12:16  * ghostbar_quit (Remote host closed the connection)
13:12:44  * ghostbarjoined
13:17:02  * ghostbarquit (Ping timeout: 240 seconds)
13:34:47  * irajoined
13:42:19  * ghostbarjoined
13:49:39  * chorrelljoined
14:01:21  * iraquit (Ping timeout: 264 seconds)
14:02:07  * AvianFluquit (Read error: Connection reset by peer)
14:02:45  * AvianFlujoined
14:09:55  * ghostbarquit (Remote host closed the connection)
14:10:24  * ghostbarjoined
14:14:46  * ghostbarquit (Ping timeout: 240 seconds)
14:42:16  <almostobsolete>What are "retries" in my job stats?
14:44:36  <chorrell>This might be useful: http://apidocs.joyent.com/manta/jobs-reference.html#failures-and-internal-retries
14:46:42  <almostobsolete>Hmmm, still not sure when it would show retries? I take it from that that it's not a problem in my code? Is there anything I can(or should) do about it?
14:48:24  * jcspjoined
14:48:25  <chorrell>good question
14:49:13  <chorrell>I'm a Joyent employee, but not one of the manta experts, someone should be on soon though who could better answer that
14:49:29  <almostobsolete>cool, thanks
14:51:15  * mcavagejoined
14:55:20  <chorrell>speaking with one of the manta dudes
14:55:37  <chorrell>it's not something you need to do anything about
14:58:08  <mcavage>almostobsolete: "retries" are an indication that we have failures internally, and are just a way for us to be transparent and give you some way of seeing "it's not you, it's us".
14:58:27  <mcavage>you don't need to take action, other than maybe to cancel a job if it's just gone off the deep end.
14:59:00  <mcavage>you would see them correlate to having lots of "InternalError" codes in mjob errors $job
15:00:18  <mcavage>almostobsolete: also, to your how to log - you can log however you want; if you want to simply capture intermediate data there is 'mtee' which saves all output data as an additional manta object. Alternatively you could log however you want (printf/syslog/...) and then have your "exec"'s last thing to do be "mput /var/log/foo" somewhere.
15:08:25  <almostobsolete>mcavage: thanks
15:09:09  <almostobsolete>Another question: quite often when I have a crashing bug in my mapper it doesn't return any error from the job, just empty output
15:11:06  <mcavage>hrm. that would likely be because it's still exiting 0 to the shell.
15:11:23  <mcavage>i would try using mlogin and running that thing to where it crashes, and see what it's exiting as.
15:11:45  <mcavage>if it's actually exiting 1 - then i would want to know more :)
15:14:08  <almostobsolete>It's a python program and it's exiting with an unhandled exception
15:14:34  <mcavage>but what's the status code to the shell?
15:14:44  <mcavage>like if you run it from an mlogin session, what's echo $?
15:14:52  <almostobsolete>just a sec
15:15:16  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
15:15:17  <almostobsolete>I'll see if I can reproduce it
15:19:22  <almostobsolete>well that's odd
15:19:33  <mcavage>?
15:19:42  <almostobsolete>the version of python running on manta returns a status of 0 when it hits an exception
15:20:14  <almostobsolete>echo "a = []; print a['yo']" > test.py; python test.py; echo $?
15:20:26  <almostobsolete>On my local machine gives 1 but on manta mlogin gives 0
15:20:37  <mcavage>really?
15:20:41  <mcavage>hmm
15:21:28  <rmustacc>Doesn't happen from mlogin.
15:21:35  <rmustacc>Is that part of a pipeline?
15:21:39  <mcavage>[email protected] # echo "a = []; print a['yo']" > test.py; python test.py; echo $?
15:21:39  <mcavage>Traceback (most recent call last):
15:21:40  <mcavage> File "test.py", line 1, in <module>
15:21:41  <mcavage> a = []; print a['yo']
15:21:43  <mcavage>TypeError: list indices must be integers, not str
15:21:45  <mcavage>1
15:21:47  <mcavage>[email protected] #
15:21:57  <almostobsolete>Oh wait no,I must have screwed up, it's giving the correct answer for me again
15:21:58  <mcavage>yeah are you doing something here where bash pipefail is biting you?
15:22:27  <almostobsolete>I am piping it into msplit
15:22:30  <almostobsolete>could that be the issue?
15:23:08  <mcavage>http://petereisentraut.blogspot.com/2010/11/pipefail.html
15:23:14  <almostobsolete>Maybe running msplit as a seperate map stage would work better?
15:23:27  <mcavage>no it's fine - this is just straight bash semantics.
15:23:43  <mcavage>bad_thing | good_thing is never going to exit 1
15:23:49  <almostobsolete>hmmm, I didn't know that
15:23:52  <mcavage>good_thing | bad_thing will exit 1
15:24:21  <almostobsolete>Ok, have stuck set -o pipefail at the top of my script
15:24:47  * irajoined
15:25:02  <mcavage>that will probably do it. be cautioned this isn't *always* what you want, which is why we don't just set it always (we have debated this internally...a lot...)
15:25:41  <almostobsolete>If I was to split my "command | msplit" into 2 seprate phases that would sort it, right?
15:26:50  <mcavage>that would, but it's going to be slower.
15:27:00  <almostobsolete>right, that makes sense
15:27:09  <mcavage>b/c you have to write all that data back to manta you would otherwise be able to just process locally.
15:28:41  <mcavage>Anyway, hope that helped - I have to drop offline for a few, but it seems rmustacc is around as well, so you're covered ;)
15:29:19  * mcavagequit (Remote host closed the connection)
15:36:34  * chorrelljoined
15:52:04  * dapjoined
16:00:45  * bcantrilljoined
16:04:35  * nfitchjoined
16:11:17  * fredkjoined
16:12:10  * nfitchquit (Quit: Leaving.)
16:14:43  * yunongjoined
16:19:08  * mcavagejoined
16:19:26  * mcavagequit (Remote host closed the connection)
16:20:17  * mcavagejoined
16:20:53  * nfitchjoined
16:23:21  * iraquit (Quit: Connection terminated.)
16:25:39  * notmatt_joined
16:43:11  * yunongquit (Quit: Leaving.)
17:03:33  * almostobsoletequit (Ping timeout: 248 seconds)
17:09:51  * bixujoined
17:13:36  * yunongjoined
17:13:46  * bixuquit (Ping timeout: 240 seconds)
17:16:23  * mamashpart
17:54:10  * bcantrillquit (Quit: Leaving.)
18:03:57  * mamashjoined
18:13:07  * bcantrilljoined
18:30:06  <bcantrill>jzu_: Yeah, sorry catching up. rmustacc knows me well. ;)
18:39:27  <jzu_>=)
18:39:37  <jzu_>bcantrill: no problemo, I was just trying my luck
19:07:27  * bixujoined
19:09:46  * mamashpart
19:09:59  * AvianFluquit (Remote host closed the connection)
19:11:46  * bixuquit (Ping timeout: 240 seconds)
19:13:17  * notmattjoined
19:13:17  * notmatt_quit (Read error: Connection reset by peer)
19:24:41  * AvianFlujoined
19:27:59  * AvianFluquit (Remote host closed the connection)
19:28:27  * AvianFlujoined
19:35:53  * bixujoined
20:17:40  * AvianFluquit (Remote host closed the connection)
20:18:09  * AvianFlujoined
20:28:58  * cburroughsquit (Ping timeout: 276 seconds)
20:58:23  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
21:24:29  * chorrelljoined
21:47:10  <mjn_>q: if i want to chunk some huge files into smaller bits as the first phase of a job, is the right approach to call mpipe w/o params multiple times in the first phase, this will result in the auto-named objects being passed to the tasks in the 2nd phase, and then i delete the temp files under /:login/jobs at the end to clean up?
21:48:19  * mjn_changed nick to mjn
21:50:31  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
21:50:52  * chorrelljoined
22:05:19  <dap>mjn_: That should work, and you wouldn't need to clean up the temporary objects. The fact that you ever see them is essentially a bug; the system should take care of those.
22:05:28  <dap>mjn: ^
22:28:03  * bixu_joined
22:30:36  * bixuquit (Ping timeout: 268 seconds)
22:33:16  <mjn>dap: ah ok, from the documentation of mpipe i wasn't sure what the intended semantics are... it *sort* of implied that "save this object" was the main use-case but also mentioned that "pipe this object to the next phase" is another use case
22:34:16  <dap>mpipe is always for sending data through the job's data stream (i.e., to the next phase, or as a job output). It's kind of a side effect that if you give the object a name, we'll save it with that name and not remove it.
22:34:26  <dap>Sounds like that should be clearer in the docs.
22:35:12  <mjn>saving it is also (in my preliminary toy experiments, anyway) a fairly important role i discovered sorta by accident, mostly b/c it fits into my mental model of a cat in|a|b|c > out filter
22:35:17  <mjn>where the final mpipe names the out
22:35:47  <mjn>while digging it out of /jobs somehow seems clunkier, esp. if it's some multi-gigabyte thing i don't want to mget
22:37:31  <mjn>it feels slightly overloaded to me that | and > are both mpipe, but that might be too much trying to shoehorn sh onto the execution model
22:41:00  * bixu_quit (Remote host closed the connection)
22:42:51  * chorrellquit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
22:47:40  * chorrelljoined
22:54:14  * bixujoined
23:01:08  <mcavage>mjn: fair. this sort of stems from the "common case" of something like mfind | mjob create 'grep foo' ^^ sort --- you just want to be able to have the stream "just work"
23:01:28  <mcavage>and have 'mget $(mjob outputs :job)' be able to resolve all that.
23:02:03  <mcavage>so it's sort of overloaded, but it's overloaded b/c that's kind of the mainline ad-hoc thing that's used. admittedly if you're naming everything and writing bigger scripts some of the trade-offs for "magic" become more distinct.
23:03:52  * chorrellquit (Quit: Textual IRC Client: www.textualapp.com)
23:04:33  * bixu_joined
23:07:57  * bixuquit (Ping timeout: 264 seconds)
23:08:23  * bixu_quit (Remote host closed the connection)
23:09:58  * bixujoined
23:11:51  * bixuquit (Remote host closed the connection)
23:27:53  <mjn>mcavage: that makes sense... i think i've been avoiding the auto-created outputs for whatever reason, maybe because it feels weird when i'm prototyping something in conceive of as running on huge files
23:28:14  <mjn>like if i do cat input | a | b | c > output and input is gigabytes, i feel like i care where 'output' ends up for good
23:29:02  <mjn>and mpipe suggested it would do (and actually does!) that, since it sends something to a named output and doesn't write the default stdout output
23:29:28  <mjn>actually maybe that's the part of the docs i found a little confusing (possibly my fault), it wasn't clear to me that mpipe suppresses output *only* when it's last, but *does* pipe when it's not last
23:31:17  <mjn>(to be clear this whole system seems pretty well conceived... i'm mostly trying to shoehorn it into my existing mental model and then reconsider whenever it doesn't fit ;-)
23:44:11  <mcavage>mjn: no worries ;) -- but yeah, i mean the workflow I/we tend to use is just to let mpipe do it's thing, and have a job under ~/jobs/...
23:44:29  <mcavage>and at least i then would do mln $(mjob outputs foo) ~/stor/my_new_home
23:44:58  <mcavage>that way you can still sanely mrm -r ~/jobs/:id and drop everything.
23:45:26  <mcavage>in the very first version of the system we actually had jobs using the same namespace, so like a job on stor/foo would result in stor/foo.1
23:45:29  <mcavage>it was terrible :)
23:48:08  <mjn>oh right, that's a part i read but didn't in any way introduce into my workflow
23:48:28  <mjn>that mln survives deletion of the original
23:49:23  <mjn>i think i still vaguely prefer explicitly specifying the output of a reduce task, but the preference is less strong since you can in effect produce an easy mv by combining mln/mrm
23:49:56  * nfitchquit (Quit: Leaving.)
23:55:10  <mcavage>well, it's up to you, really :)
23:55:19  <mcavage>like unix, there's N ways to do the same thing.
23:55:40  * bixujoined