00:07:42  <nahamu>I'd love a flag to mput that means "keep trying until you succeed"
00:08:11  <nahamu>or at least "try n times before giving up"
00:12:07  * therealkoopajoined
00:12:28  <dap_>I thought it did try 3 times in recent versions
00:12:54  <nahamu>oh, maybe I need to update the version in my SmartOS build zone...
00:13:55  <dap_>I could be wrong
00:23:52  <trentm>yes, 3 retries: https://github.com/joyent/node-manta/blob/master/bin/mput#L259-L265
00:24:01  <nahamu>yeah, I appear to have sdc-manta-1.2.6 installed.
00:24:13  <trentm>sdc-manta ?
00:24:15  <nahamu>npm install manta has obtained 1.4.6
00:24:33  <nahamu>a pkgsrc packge that somehow ended up in my build zone.
00:24:40  <trentm>oh, I see
00:29:14  <nahamu>well, now I've hardcoded my build script to use that newer version. hopefully that will clear up the occasional upload failures I was seeing.
00:29:31  * lloyddequit (Remote host closed the connection)
00:29:54  <nahamu>thanks for the tip dap_ and trentm!
00:32:38  <_Tenchi_>know why the uploads were failing ?
00:34:56  * sigxcpuquit (Quit: sigxcpu)
00:38:40  * therealkoopaquit (Remote host closed the connection)
00:51:52  <nahamu>one got a connection reset IIRC
00:52:11  <nahamu>that shell is busy doing another build so I've lost the error message.
00:52:43  * therealkoopajoined
00:57:20  <_Tenchi_>local problem?
00:59:24  <nahamu>I'm not sure.
01:00:24  * lloyddejoined
01:00:47  * ryancnelsonquit (Quit: Leaving.)
01:02:22  * therealkoopaquit (Remote host closed the connection)
01:05:03  * lloyddequit (Ping timeout: 250 seconds)
01:05:54  * therealkoopajoined
01:06:24  * trentmquit (Quit: Leaving.)
01:09:31  <nahamu>"mput: Error: socket hang up"
01:09:37  <_Tenchi_>ouch
01:09:55  <nahamu>doesn't appear to me to have retried, but I wasn't watching it, I just found it failed.
01:15:03  <nahamu>okay, I'm going to just wrap that in a shell function that will just keep retrying and sleeping for a second after each failure...
01:15:21  <dap_>If you're piping to mput, it can't retry
01:15:26  <dap_>It can only retry using "mput -f"
01:15:53  <nahamu>I'm doing it with -f
01:18:36  <nahamu>https://paste.ec/paste/+OMCGwlF#gHgznZmdlUtCWB4Z7r2CUKvQ6Ge31OwVI8oSh4bK+qe
01:18:39  * trentmjoined
01:20:00  * ed209quit (Remote host closed the connection)
01:20:07  * ed209joined
01:27:39  * trentmquit (Quit: Leaving.)
01:28:22  * dap_quit (Quit: Leaving.)
01:29:30  <nahamu>It does appear to be retrying. I'm now seeing the progress bar doing resets.
02:01:08  * lloyddejoined
02:03:15  * fredkquit (Quit: Leaving.)
02:05:24  * lloyddequit (Ping timeout: 244 seconds)
02:38:55  * therealkoopaquit (Remote host closed the connection)
02:47:03  * kapil__joined
02:47:29  * therealkoopajoined
02:52:15  * therealkoopaquit (Ping timeout: 264 seconds)
02:55:29  * namtziglajoined
02:55:40  * therealkoopajoined
03:00:16  * therealkoopaquit (Ping timeout: 265 seconds)
03:02:02  * lloyddejoined
03:06:21  * lloyddequit (Ping timeout: 252 seconds)
03:09:55  * therealkoopajoined
03:16:42  * xmerlin_joined
03:17:52  * xmerlinquit (Ping timeout: 252 seconds)
03:19:22  * therealkoopaquit (Ping timeout: 240 seconds)
03:25:23  * therealkoopajoined
03:26:43  * pmooney_joined
03:29:37  * pmooneyquit (Ping timeout: 256 seconds)
03:31:22  * therealkoopaquit (Ping timeout: 240 seconds)
03:33:58  * therealkoopajoined
03:38:33  * therealkoopaquit (Ping timeout: 264 seconds)
03:59:11  * therealkoopajoined
04:02:49  * lloyddejoined
04:04:17  * dobsonquit (Ping timeout: 252 seconds)
04:07:19  * lloyddequit (Ping timeout: 245 seconds)
04:08:28  * therealkoopaquit (Ping timeout: 252 seconds)
04:16:02  * trentmjoined
04:16:34  * dobsonjoined
04:22:15  * dobsonquit (Ping timeout: 252 seconds)
04:34:02  * namtziglaquit (Ping timeout: 245 seconds)
04:40:19  * therealkoopajoined
04:42:49  * yunong_joined
04:43:06  * yunongquit (Read error: Connection reset by peer)
04:44:37  * therealkoopaquit (Ping timeout: 252 seconds)
04:45:54  * marsellquit (Quit: marsell)
04:46:25  * therealkoopajoined
04:52:33  * therealkoopaquit (Ping timeout: 250 seconds)
05:03:30  * lloyddejoined
05:05:15  * lloydde_joined
05:07:57  * lloyddequit (Ping timeout: 264 seconds)
05:09:55  * lloydde_quit (Ping timeout: 256 seconds)
05:30:43  * lloyddejoined
05:31:53  * namtziglajoined
05:35:09  * lloyddequit (Ping timeout: 250 seconds)
05:36:06  * namtziglaquit (Ping timeout: 252 seconds)
05:46:38  * nfitchquit (Ping timeout: 245 seconds)
05:58:55  * nfitchjoined
06:00:02  * trentmquit (Quit: Leaving.)
06:23:28  * dobsonjoined
06:31:24  * lloyddejoined
06:33:01  * therealkoopajoined
06:33:11  * namtziglajoined
06:33:12  * dobsonquit (Ping timeout: 246 seconds)
06:35:37  * trentmjoined
06:36:19  * lloyddequit (Ping timeout: 265 seconds)
06:37:22  * namtziglaquit (Ping timeout: 240 seconds)
06:38:07  * dobsonjoined
06:39:07  * trentmquit (Client Quit)
06:41:13  * therealkoopaquit (Ping timeout: 252 seconds)
06:45:15  * dobsonquit (Ping timeout: 252 seconds)
06:48:14  * dobsonjoined
06:55:28  * therealkoopajoined
06:59:47  * therealkoopaquit (Ping timeout: 246 seconds)
07:02:08  * therealkoopajoined
07:06:46  * therealkoopaquit (Ping timeout: 265 seconds)
07:32:13  * lloyddejoined
07:33:51  * namtziglajoined
07:36:55  * lloyddequit (Ping timeout: 250 seconds)
07:38:33  * namtziglaquit (Ping timeout: 264 seconds)
08:15:04  * therealkoopajoined
08:24:33  * therealkoopaquit (Ping timeout: 245 seconds)
08:25:26  * pgalejoined
08:33:08  * lloyddejoined
08:34:41  * namtziglajoined
08:37:35  * lloyddequit (Ping timeout: 250 seconds)
08:39:03  * namtziglaquit (Ping timeout: 264 seconds)
08:44:49  * therealkoopajoined
08:49:35  * therealkoopaquit (Ping timeout: 250 seconds)
09:20:48  * marselljoined
09:33:45  * lloyddejoined
09:33:51  * sigxcpujoined
09:36:12  * namtziglajoined
09:38:32  * lloyddequit (Ping timeout: 265 seconds)
09:40:27  * namtziglaquit (Ping timeout: 246 seconds)
09:52:37  * ed209quit (Ping timeout: 252 seconds)
09:53:21  * |woody|quit (Ping timeout: 252 seconds)
09:59:17  * |woody|joined
10:01:25  * therealkoopajoined
10:04:55  * dobsonquit (Ping timeout: 255 seconds)
10:10:15  * therealkoopaquit (Ping timeout: 256 seconds)
10:34:34  * lloyddejoined
10:36:52  * namtziglajoined
10:38:47  * lloyddequit (Ping timeout: 250 seconds)
10:41:22  * namtziglaquit (Ping timeout: 265 seconds)
10:47:12  * therealkoopajoined
10:53:19  * therealkoopaquit (Ping timeout: 256 seconds)
10:55:52  * therealkoopajoined
11:00:25  * therealkoopaquit (Ping timeout: 264 seconds)
11:14:25  * therealkoopajoined
11:15:50  * sigxcpuquit (Quit: sigxcpu)
11:19:51  * therealkoopaquit (Ping timeout: 264 seconds)
11:35:14  * lloyddejoined
11:37:45  * namtziglajoined
11:40:01  * lloyddequit (Ping timeout: 264 seconds)
11:43:03  * namtziglaquit (Ping timeout: 250 seconds)
11:59:29  * therealkoopajoined
12:05:04  * therealkoopaquit (Ping timeout: 255 seconds)
12:13:48  * jperkin_joined
12:19:11  * jperkinquit (*.net *.split)
12:29:06  * therealkoopajoined
12:36:01  * lloyddejoined
12:38:18  * sigxcpujoined
12:39:23  * namtziglajoined
12:40:23  * lloyddequit (Ping timeout: 244 seconds)
12:43:43  * namtziglaquit (Ping timeout: 245 seconds)
12:47:00  * ed209joined
13:04:07  * ffahimiquit (Remote host closed the connection)
13:20:49  * kapil__quit (Quit: Connection closed for inactivity)
13:36:50  * lloyddejoined
13:40:02  * namtziglajoined
13:41:22  * lloyddequit (Ping timeout: 255 seconds)
13:45:17  * namtziglaquit (Ping timeout: 245 seconds)
14:37:29  * lloyddejoined
14:41:39  * namtziglajoined
14:41:46  * lloyddequit (Ping timeout: 245 seconds)
14:46:00  * namtziglaquit (Ping timeout: 246 seconds)
14:51:01  * jperkin_changed nick to jperkin
15:27:18  * namtziglajoined
15:38:21  * lloyddejoined
15:40:03  * ffahimijoined
15:43:07  * lloyddequit (Ping timeout: 250 seconds)
15:48:08  * chorrelljoined
16:10:50  * ffahimiquit (Remote host closed the connection)
16:14:11  * nfitchquit (Ping timeout: 250 seconds)
16:22:21  * xmerlin_quit (Quit: Sto andando via)
16:26:05  * nfitchjoined
16:29:51  * namtziglaquit (Ping timeout: 265 seconds)
16:39:16  * lloyddejoined
16:43:31  * lloyddequit (Ping timeout: 256 seconds)
16:51:15  * namtziglajoined
17:01:04  * trentmjoined
17:03:26  * fredkjoined
17:13:02  * trentmquit (Quit: Leaving.)
17:21:17  * dap_joined
17:24:10  * dobsonjoined
17:26:55  * trentmjoined
17:36:20  * ryancnelsonjoined
17:39:56  * lloyddejoined
17:44:34  * lloyddequit (Ping timeout: 264 seconds)
18:00:39  * dobsonquit (Ping timeout: 252 seconds)
18:08:54  * pgalequit (Quit: Leaving.)
18:12:12  * dobsonjoined
18:13:16  * pmooney_changed nick to pmooney
18:27:47  * dobsonquit (Ping timeout: 252 seconds)
18:39:15  * dobsonjoined
18:40:40  * lloyddejoined
18:45:10  * lloyddequit (Ping timeout: 264 seconds)
18:54:26  * dobsonquit (Quit: Leaving)
19:02:43  * pmooneyquit (Quit: coffee)
19:05:48  <namtzigla>hi everyone
19:06:20  <namtzigla>I want to report a bug on manta-init
19:06:40  <namtzigla>where should I do that ?
19:07:33  <namtzigla>and what additional info is required ?
19:07:57  <nahamu>probably https://github.com/joyent/manta/issues
19:08:04  <rmustacc>https://github.com/joyent/sdc-manta/issues
19:08:29  <nahamu>ignore me. :)
19:18:30  * pgalejoined
19:19:15  * pmooneyjoined
19:19:46  * namtziglaquit (Remote host closed the connection)
19:20:02  * namtziglajoined
19:21:50  <namtzigla>can someone help me with my manta installation? after I successfully install it it does not seems to have any process running, for example in the postgres node there is no postgres process running or a tcp port in LISTEN mode
19:24:40  <dap_>namtzigla: are any services in maintenance?
19:25:39  <namtzigla>no, sdc ops portal show them as running
19:26:22  <namtzigla>this is the config and the output of some commands that I've run there :https://gist.github.com/namtzigla/719a61a64d0e3eb41146
19:27:10  <dap_>Sorry, I meant SMF services. From a GZ, "svcs -Zxv" will show you the SMF services in maintenance on the whole physical server.
19:29:07  <namtzigla>yeah
19:29:09  <namtzigla>all of them
19:29:55  <dap_>well, that command only prints the ones in maintenance. but it sounds like there are a lot?
19:30:15  <namtzigla>I just update the gist
19:30:22  <namtzigla>I think they are all of them
19:30:45  <namtzigla>or 13 of them
19:31:17  <dap_>I'd probably start with 1.postgres.LAX.sdc.lax-3ef849d8
19:32:08  <dap_>mdata:execute is the service responsible for setting up the zone the first time it's booted. Failures in this service often reflect a configuration problem, but unfortunately they're usually not very clear about it.
19:32:26  <dap_>The log file there (/zones/3ef849d8-65d5-454a-b317-662c29ecc7cc/root/var/svc/log/smartdc-mdata:execute.log) will have bash xtrace output for the setup script
19:32:37  <dap_>The usual next step is to go through that and figure out what went wrong.
19:32:58  <dap_>It's pretty likely that either it affected the other zones' mdata:execute services as well, or those failed because other zones on which they depend were also broken.
19:33:41  <namtzigla>ok, thanks I will try to figure out
19:34:20  <dap_>Let me know if you need more help with it.
19:34:26  <namtzigla>I think zk is down
19:34:26  <namtzigla>[[2015-02-06T03:37:37Z] /opt/smartdc/boot/scripts/util.sh:146: manta_ensure_zk(): nc -w 1 10.31.7.13 2181 [ Feb 6 03:39:22 Method or service exit timed out. Killing contract 945. ] [ Feb 6 03:39:22 Method "start" failed due to signal KILL. ]
19:34:30  <namtzigla>this is the last entry
19:36:36  <dap_>Yeah, that seems likely, and that'll affect many setup scripts. Do you know if that IP is a correct IP for one of the Manta nameservice zones?
19:36:38  <dap_>You can use:
19:36:43  <namtzigla>it seems that I should have 6 zookeeper servers up but I don't have any
19:36:45  <dap_>manta-adm show -o zonename,primary_ip nameservice
19:37:01  <dap_>You should generally have 3 or 5.
19:37:13  <namtzigla>well ... I have 1
19:37:22  <dap_>What's that "manta-adm show" say?
19:37:28  <namtzigla>ZONENAME PRIMARY IP cf114ba8-bac6-4d3b-b66e-5a32563af92b 10.31.7.17
19:37:49  <dap_>You have one because the manta-adm config file only has one "nameservice" zone. (ZK is the "nameservice" zone.)
19:38:04  <dap_>It looks like you must have provisioned a nameservice zone previously, and then removed it
19:38:49  <namtzigla>well ... I did multiple attempts
19:38:51  <dap_>Or alternatively, you had a "nameservice" zone provision fail
19:39:24  <namtzigla>is there a way to update the config manually ?
19:41:19  <dap_>Yes, but it's unfortunately pretty manual. FYI, the ticket for improving that is https://smartos.org/bugview/MANTA-2477
19:41:25  * lloyddejoined
19:41:40  <dap_>and the ticket for the problem you ran into is: https://smartos.org/bugview/MANTA-2175
19:42:18  * namtzigla_joined
19:42:27  <namtzigla_>sorry I got cutoff
19:42:56  <dap_>I said: Yes, but it's unfortunately pretty manual.  FYI, the ticket for improving that is https://smartos.org/bugview/MANTA-2477. and the ticket for the problem you ran into is: https://smartos.org/bugview/MANTA-2175.
19:42:59  <dap_>So here's the crash course: Manta stores metadata like this inside an SDC service called SAPI.
19:43:14  <dap_>I'm going to give you a command to *show* that metadata, but do NOT paste it here or put it into the gist because it contains a private key...
19:43:27  <dap_>sdc-sapi /applications?name=manta
19:43:31  <namtzigla_>thanks
19:43:34  <dap_>You run that from the GZ of the headnode, and it will print the metadata.
19:43:45  <dap_>The thing you need to modify is the ZK_SERVERS block.
19:44:10  <dap_>You probably have two servers in there: 10.31.7.17 and 10.31.7.13. You want to remove the block for 10.31.7.13.
19:44:46  <dap_>Actually, how painful would it be for you to remove all the Manta zones and start again?
19:45:07  * namtziglaquit (Ping timeout: 265 seconds)
19:45:22  <namtzigla_>not bad
19:45:23  <dap_>That's definitely the easiest: run manta-factoryreset, manta-init, and deploy again. You won't have to download images again, but you will have to regenerate the consistent hash ring.
19:45:42  <namtzigla_>well, b/c I'm on production mode I can't run manta-factoryreset
19:45:54  <namtzigla_>the script does not allow it
19:46:15  * lloyddequit (Ping timeout: 264 seconds)
19:47:21  <dap_>D'oh. Well, you can remove that check…. or you can try updating in place. If you want to update in place, you'd use "sapiadm update" to update the ZK_SERVERS property. Once you've done that, you'll want to reboot all of the zones associated with Manta, probably in dependency order (postgres, then moray, then electric moray, and then it probably doesn't matter too much).
19:47:51  <namtzigla_>ok
19:47:57  <namtzigla_>I will try the 2nd path
19:48:12  <namtzigla_>I will learn something new
19:48:13  <namtzigla_>thanks
19:49:17  <dap_>No problem. Sorry for the pain.
19:50:51  <namtzigla_>no pain, just learning :)
19:51:06  * dobsonjoined
19:53:08  * pgalequit (Quit: Leaving.)
19:55:04  * dobsonquit (Excess Flood)
20:03:52  * dobsonjoined
20:12:52  * chorrellquit (Ping timeout: 255 seconds)
20:13:47  * chorrelljoined
20:13:59  * chorrellquit (Changing host)
20:13:59  * chorrelljoined
20:20:00  * ed209quit (Remote host closed the connection)
20:20:07  * ed209joined
20:20:21  * dobsonquit (Ping timeout: 252 seconds)
20:20:56  * pmooney_joined
20:20:57  * pmooneyquit (Read error: Connection reset by peer)
20:27:14  * dobsonjoined
20:42:03  * dobsonquit (Ping timeout: 245 seconds)
20:42:10  * lloyddejoined
20:46:38  * lloyddequit (Ping timeout: 245 seconds)
21:09:22  * namtzigla_quit (Ping timeout: 240 seconds)
21:28:59  * namtziglajoined
21:42:56  * lloyddejoined
21:47:19  * lloyddequit (Ping timeout: 245 seconds)
21:48:14  * chorrellquit (Quit: My Mac has gone to sleep. ZZZzzz…)
22:00:22  * pmooney_quit (Quit: WeeChat 1.1.1)
22:03:30  * pmooneyjoined
22:04:37  * dap_1joined
22:06:31  * dap_quit (Ping timeout: 256 seconds)
22:12:50  * therealkoopaquit (Remote host closed the connection)
22:31:32  * therealkoopajoined
22:37:40  <namtzigla>hi, does anyone know how I can change the postgres db ip in manta instalation ?
22:38:35  <namtzigla>and moray try to connect to the old ip
22:43:53  * lloyddejoined
22:48:01  * lloyddequit (Ping timeout: 245 seconds)
22:57:59  * therealkoopaquit (Remote host closed the connection)
23:06:36  <dap_1>moray should find the IP from the IP that postgres registered itself with
23:18:20  * therealkoopajoined
23:26:33  * chorrelljoined
23:44:33  * lloyddejoined
23:48:26  * namtziglaquit (Ping timeout: 244 seconds)
23:49:11  * lloyddequit (Ping timeout: 246 seconds)
23:54:46  * pmooneyquit (Quit: WeeChat 1.1.1)