Performance measurement

Hi.

In this post you can see some of the performed measurements, and its results.
Note: please use a browser which supports the <details> tag.

Measurement #1 (based on Viktor Tusa's idea):

Goal:

To find out, if the amqp-source is slower than the amqp-destination.

Structure:

reading from a file -> amqp-destination -> message broker -> amqp-source -> write to a file (e.g. /dev/null)

Question:

Does the messages pile up in a queue in the message broker, or not?

syslog-ng.conf

#syslog-ng.conf
@version: 3.5

source s_amqp {   
    amqp(
        exchange("syslog-ng")
        host("localhost")
        username("guest")
        password("guest")
        routing_key("secret")
        vhost("/")
    );
};

destination d_amqp {
    amqp(
        exchange("syslog-ng")
        host("localhost")
        body("$MESSAGE")
        username("guest")
        exchange_type("header")
        password("guest")
        routing_key("secret")
        value-pairs(
            scope("selected-macros" "nv-pairs" "sdata")
 )
    );
};

source s_local {
    file("/tmp/sock");
};

destination d_file {
    file("/dev/null" template("$DATE $(format-json --scope nv-pairs)\n"));
};

log {
    source(s_local);
    destination(d_amqp);
};

log {
    source(s_amqp);
    destination(d_file);
};

Little script that writes to a file periodically

#!/bin/bash
while true
do
    echo "Message" >> /tmp/sock
done

Results:

tl;dr The destination generates ~2k-8k more messages per second than the source can handle.

State of the queues

Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308444139
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308445129
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308446100
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308447103
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308448094
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308449081
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 0
...done.
1408308449832
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 31
...done.
1408308450757
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 159
...done.
1408308451662
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 4371
...done.
1408308452447
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 9043
...done.
1408308453370
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 14965
...done.
1408308454281
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 23287
...done.
1408308455155
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 29327
...done.
1408308455975
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 31894
...done.
1408308456793
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 39888
...done.
1408308457716
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 46319
...done.
1408308458477
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 51781
...done.
1408308459384
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 59519
...done.
1408308460179
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 66341
...done.
1408308461093
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 75777
...done.
1408308462064
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 81866
...done.
1408308462978
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 87513
...done.
1408308463748
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 96992
...done.
1408308464921
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 99337
...done.
1408308465911
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 108929
...done.
1408308466884
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 118302
...done.
1408308467891
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 122037
...done.
1408308469011
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 127546
...done.
1408308469858
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 139121
...done.
1408308470795
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 142632
...done.
1408308471913
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 151926
...done.
1408308472886
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 160832
...done.
1408308473723
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 162968
...done.
1408308475047
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 171215
...done.
1408308475925
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 178404
...done.
1408308476717
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 182182
...done.
1408308477543
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 189568
...done.
1408308478443
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 197558
...done.
1408308479529
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 206171
...done.
1408308480359
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 207516
...done.
1408308482155
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 215089
...done.
1408308483532
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 227599
...done.
1408308484416
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 228505
...done.
1408308485251
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 240284
...done.
1408308486157
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 248033
...done.
1408308486958
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 255792
...done.
1408308488189
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 265552
...done.
1408308489191
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 275802
...done.
1408308490023
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 285169
...done.
1408308490935
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 293859
...done.
1408308491882
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 294042
...done.
1408308492625
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 299697
...done.
1408308493604
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 305819
...done.
1408308494611
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 309480
...done.
1408308495383
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 312242
...done.
1408308496245
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 316079
...done.
1408308497271
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 319693
...done.
1408308498016
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 322427
...done.
1408308498770
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 325501
...done.
1408308499622
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 329436
...done.
1408308500483
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 332937
...done.
1408308501339
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 336978
...done.
1408308502287
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 337352
...done.
1408308504064
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 340965
...done.
1408308505013
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 344697
...done.
1408308505971
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 349119
...done.
1408308506806
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 352022
...done.
1408308507736
Listing queues ...
amq.gen-9kULCCRKPZ1ErYqIW3GI9A 355157
...done.
1408308508478

Update: According to the latest discovery, sometimes the worker thread falls asleep, but it doesn't wake up. Because of this, we can assume, that the amqp-source is probably not a bottleneck.

I could not pile up the messages with this little program.

amqp_producer.c

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include <stdint.h>
#include <unistd.h>
#include <time.h>
#include <amqp_tcp_socket.h>
#include <amqp.h>
#include <amqp_framing.h>

int main(int argc, char const * const * argv)
{
    char const *            hostname;
    int                     port, status;
    char const *            exchange;
    char const *            routingkey;
    char const *            messagebody;
    amqp_socket_t *         socket = NULL;
    amqp_connection_state_t conn;

    if (argc < 6) {
        fprintf(stderr, "Usage: amqp_sendstring host port exchange routingkey messagebody\n");
        return 1;
    }

    hostname    = argv[1];
    port        = atoi(argv[2]);
    exchange    = argv[3];
    routingkey  = argv[4];
    messagebody = argv[5];

    conn        = amqp_new_connection( );

    socket      = amqp_tcp_socket_new(conn);

    status      = amqp_socket_open(socket, hostname, port);

    amqp_login(conn, "/", 0, 131072, 0, AMQP_SASL_METHOD_PLAIN, "guest", "guest");
    amqp_channel_open(conn, 1);
    amqp_get_rpc_reply(conn);

    amqp_exchange_declare(conn, 1, amqp_cstring_bytes(exchange), amqp_cstring_bytes("direct"), 0, 0, amqp_empty_table);

    while (1) {
        amqp_basic_properties_t props;
        props._flags        = AMQP_BASIC_CONTENT_TYPE_FLAG | AMQP_BASIC_DELIVERY_MODE_FLAG | AMQP_BASIC_TIMESTAMP_FLAG | AMQP_BASIC_HEADERS_FLAG;
        props.content_type  = amqp_cstring_bytes("text/plain");
        props.delivery_mode = 2; /* persistent delivery mode */
        props.timestamp     = time(NULL);

        amqp_bytes_t         keybytes   = { .len = 9, .bytes = "kukifasz" };
        amqp_bytes_t         valuebytes = { .len = 9, .bytes = "WEHEHEHE" };
        amqp_field_value_t   value      = { .kind = AMQP_FIELD_KIND_BYTES, .value.bytes = valuebytes };
        amqp_table_entry_t * entries    = (amqp_table_entry_t *)malloc(sizeof( amqp_table_entry_t ));
        entries[0].key   = keybytes;
        entries[0].value = value;
        amqp_table_t table = { .num_entries = 1, .entries = entries };
        props.headers    = table;

        amqp_basic_publish(conn, 1, amqp_cstring_bytes(exchange), amqp_cstring_bytes(routingkey),
                           0,
                           0,
                           &props,
                           amqp_cstring_bytes(messagebody));
    }

    amqp_channel_close(conn, 1, AMQP_REPLY_SUCCESS);
    amqp_connection_close(conn, AMQP_REPLY_SUCCESS);
    amqp_destroy_connection(conn);
    return 0;
}

Measurement #2:

Goal:

To measure the consumption speed of the amqp-source.

Idea:

Pile up a lot of messages in a queue, start the consumption. The ratio of the number of the consumed messages and the elapsed time will give us the desired speed.

Result:

tl;dr The amqp-source can process ~6.5k messages per second.

much data; very raw; wow

Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362137093
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362138147
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362139136
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362140204
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362141295
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362142373
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362143376
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 646997
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362144457
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 644707
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362145810
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 628786
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362146646
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 610098
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362147468
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 603053
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362148325
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 595089
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362149261
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 584890
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362150129
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 574289
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362150963
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 571439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362151756
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 568880
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362152622
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 564339
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362153427
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 562189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362154202
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 558389
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362155033
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 554239
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362155887
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 550039
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362156675
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 545139
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362157522
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 540989
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362158336
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 536539
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362159151
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 532289
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362159954
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 528439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362160731
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 524262
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362161543
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 519702
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362162326
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 514189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362163152
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 508739
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362163982
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 504289
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362164743
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 499089
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362165591
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 496789
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362166544
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 492689
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362167403
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 488839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362168289
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 484189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362169113
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 479439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362169909
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 474367
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362170707
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 469489
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362171476
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 464439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362172243
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 459839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362173028
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 455111
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362173792
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 449489
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362174597
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 443989
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362175429
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 439589
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362176223
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 434539
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362177030
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 431039
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362177765
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 427839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362178512
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 423189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362179323
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 418189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362180116
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 412839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362180917
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 407636
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362181705
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 403792
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362182489
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 398478
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362183333
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 393139
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362184096
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 388051
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362184894
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 382889
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362185659
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 377639
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362186437
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 372239
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362187259
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 366839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362188064
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 362239
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362188881
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 357550
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362189655
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 352339
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362190474
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 346939
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362191286
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 341689
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362192070
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 336839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362192841
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 332639
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362193636
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 327389
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362194461
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 322889
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362195253
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 317789
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362196026
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 313089
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362196797
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 308139
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362197566
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 302939
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362198349
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 298239
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362199130
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 293299
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362199923
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 289221
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362200665
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 284139
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362201438
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 278589
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362202292
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 273389
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362203115
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 268439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362203967
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 264239
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362204802
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 259339
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362205672
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 254697
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362206482
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 249427
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362207290
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 244739
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362208093
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 239939
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362208882
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 234639
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362209746
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 229639
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362210571
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 226041
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362211395
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 221039
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362212158
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 216289
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362212942
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 211346
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362213811
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 206789
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362214655
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 201906
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362215466
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 196739
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362216258
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 192289
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362217023
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 187289
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362217814
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 182039
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362218637
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 177439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362219466
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 172622
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362220256
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 168339
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362221024
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 163189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362221818
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 158989
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362222611
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 153691
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362223451
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 148339
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362224262
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 144339
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362225026
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 138539
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362225907
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 134439
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362226721
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 129043
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362227567
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 124089
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362228398
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 118839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362229181
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 113789
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362229965
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 108889
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362230801
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 103639
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362231588
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 98539
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362232451
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 93939
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362233282
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 88589
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362234090
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 83239
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362234904
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 78358
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362235694
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 72939
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362236483
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 68489
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362237264
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 63489
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362238063
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 59139
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362238830
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 53739
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362239629
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 48785
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362240401
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 43839
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362241190
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 38591
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362241970
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 34089
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362242797
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 29189
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362243610
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 17057
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362244483
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 5273
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362245376
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362246132
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362247018
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362248030
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362249092
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362250091
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362251132
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362252207
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362253216
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362254239
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362255307
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362256333
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362257358
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362258400
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362259476
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362260544
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362261584
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362262650
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362263710
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362264774
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362265944
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362266975
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362268011
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362269080
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362270125
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362271247
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362272313
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362273301
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362274323
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362275334
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362276324
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362277406
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362278478
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362279570
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362280593
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362281647
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362282675
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362283792
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362284816
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362285929
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362286998
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362287994
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362288957
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362289985
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362290964
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362291986
Listing queues ...
amq.gen-Gs0Kod1ReFUHtRi9GuC3jw 529
amq.gen-iW0FWZ2o6uBK8gid8mB5Wg 0
amq.gen-pA3hJrUuKYnUfIo6wkL9eQ 339
amq.gen-ym--ZIx9YhIK1LyYQn6AKQ 2245
...done.
1408362293063

Note: Structure of an entry: output of `rabbitmqctl list_queues`; current timestamp in milliseconds

Measurement #3

Goal:

To measure the message production speed of the message generator tool.
Note: The purpose of this measurement is to get the speed of a more simple program. If we have this result, we can compare it to the speed of amqp-source.

Result:

tl;dr The message generator tool can produce ~8000 messages per second on my machine.

Raw data of the measurement

Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 0
...done.
1408367438682
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 16
...done.
1408367439899
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 92
...done.
1408367440977
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 2
...done.
1408367441992
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 2572
...done.
1408367442914
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 9651
...done.
1408367443824
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 14842
...done.
1408367444735
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 22248
...done.
1408367445695
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 28387
...done.
1408367446611
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 35821
...done.
1408367447504
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 42782
...done.
1408367448396
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 43911
...done.
1408367449440
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 55101
...done.
1408367450347
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 66798
...done.
1408367451242
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 77142
...done.
1408367452177
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 87520
...done.
1408367453428
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 92536
...done.
1408367454357
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 104898
...done.
1408367455323
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 107927
...done.
1408367456451
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 119589
...done.
1408367457380
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 128095
...done.
1408367458203
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 138104
...done.
1408367459728
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 152050
...done.
1408367460861
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 162994
...done.
1408367461779
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 173464
...done.
1408367463558
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 182973
...done.
1408367464521
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 193464
...done.
1408367465478
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 201575
...done.
1408367466653
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 211971
...done.
1408367467633
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 220750
...done.
1408367468695
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 235873
...done.
1408367469667
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 239001
...done.
1408367471633
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 245574
...done.
1408367472796
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 251877
...done.
1408367473956
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 258870
...done.
1408367474993
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 269531
...done.
1408367475983
Listing queues ...
amq.gen-pp7sCiet1TxLWACVEUv-Wg 278593
...done.
1408367476997

Note: The structure is the same as in Measurement #2.

Comparing the results of the last two measurements, I am quite happy with the performance of the amqp-source.

How to transform an ordinary source into a threaded source?

Why should we use a different thread for a source module?

We can imagine such situations, when creating a message in a source module can be a costly operation, and if this module runs in the same thread as the main event handler then the whole syslog-ng became unresponsive therefore losing performance and may or may not misses occured events.

In these situations it is adviced to create an other thread for this source module.
However it can be tought that it introduces some typical problems of a multi-threaded problem, this is not as much pain as we anticipated.

Example (original file):

static void
trigger_source_triggered (gpointer s)
{
  TriggerSource *self = (TriggerSource *) s;
  LogMessage *msg;
  LogPathOptions path_options = LOG_PATH_OPTIONS_INIT;

  // An operation that needs a lot of time
  sleep(2); 
  main_loop_assert_main_thread ();

  if (!log_source_free_to_send (&self->super))
    goto end;

  msg = log_msg_new_internal (LOG_INFO, self->options->message);
  path_options.ack_needed = FALSE;

  log_pipe_queue (&self->super.super, msg, &path_options);

 end:
  trigger_source_update_watches (self);
}

With this modification you can check whether syslog-ng become blocked.

Transformation example:

First of all, we have to get rid of the iv_timer part (in this example we do not need it):

/*
* Copyright (c) 2013 BalaBit IT Ltd, Budapest, Hungary
* Copyright (c) 2013 Gergely Nagy 
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License version 2 as published
* by the Free Software Foundation, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
*
* As an additional exemption you are allowed to compile & link against the
* OpenSSL libraries as published by the OpenSSL project. See the file
* COPYING for details.
*
*/

#include "trigger-source.h"

#include "driver.h"
#include "logsource.h"
#include "mainloop.h"

typedef struct
{
  gint trigger_freq;
  gchar *message;
} TriggerOptions;

typedef struct _TriggerSourceDriver
{
  LogSrcDriver super;
  LogSource *source;
  LogSourceOptions source_options;

  TriggerOptions options;
} TriggerSourceDriver;

typedef struct
{
  LogSource super;

  TriggerOptions *options;
} TriggerSource;

#ifndef SCS_TRIGGER
#define SCS_TRIGGER 0
#endif

/*
* TriggerSource
*/

static void
trigger_source_triggered (gpointer s)
{
  TriggerSource *self = (TriggerSource *) s;
  LogMessage *msg;
  LogPathOptions path_options = LOG_PATH_OPTIONS_INIT;

  main_loop_assert_main_thread ();

  if (!log_source_free_to_send (&self->super))
    goto end;

  msg = log_msg_new_internal (LOG_INFO, self->options->message);
  path_options.ack_needed = FALSE;

  log_pipe_queue (&self->super.super, msg, &path_options);

 end:
}

static gboolean
trigger_source_init (LogPipe *s)
{
  TriggerSource *self = (TriggerSource *)s;

  if (!log_source_init (s))
    return FALSE;

  iv_validate_now ();

  return TRUE;
}

static gboolean
trigger_source_deinit (LogPipe *s)
{
  TriggerSource *self = (TriggerSource *)s;
  return log_source_deinit (s);
}


static LogSource *
trigger_source_new (TriggerSourceDriver *owner, LogSourceOptions *options)
{
  TriggerSource *self = g_new0 (TriggerSource, 1);

  log_source_init_instance (&self->super);
  log_source_set_options (&self->super, options, 0, SCS_TRIGGER,
                          owner->super.super.id, NULL, FALSE);

  self->options = &owner->options;

  self->super.super.init = trigger_source_init;
  self->super.super.deinit = trigger_source_deinit;

  return &self->super;
}

/*
* TriggerSourceDriver
*/

static gboolean
trigger_sd_init (LogPipe *s)
{
  TriggerSourceDriver *self = (TriggerSourceDriver *)s;
  GlobalConfig *cfg = log_pipe_get_config (s);

  if (!log_src_driver_init_method (s))
    return FALSE;

  if (self->options.trigger_freq <= 0)
    self->options.trigger_freq = 10;

  if (!self->options.message)
    self->options.message = g_strdup ("Trigger source is trigger happy.");

  log_source_options_init (&self->source_options, cfg, self->super.super.group);
  self->source = trigger_source_new (self, &self->source_options);

  log_pipe_append (&self->source->super, s);
  log_pipe_init (&self->source->super, cfg);

  return TRUE;
}

static gboolean
trigger_sd_deinit (LogPipe *s)
{
  TriggerSourceDriver *self = (TriggerSourceDriver *)s;

  if (self->source)
    {
      log_pipe_deinit (&self->source->super);
      log_pipe_unref (&self->source->super);
      self->source = NULL;
    }

  g_free (self->options.message);

  return log_src_driver_deinit_method (s);
}

LogSourceOptions *
trigger_sd_get_source_options (LogDriver *s)
{
  TriggerSourceDriver *self = (TriggerSourceDriver *)s;

  return &self->source_options;
}

void
trigger_sd_set_trigger_freq (LogDriver *s, gint freq)
{
  TriggerSourceDriver *self = (TriggerSourceDriver *)s;

  self->options.trigger_freq = freq;
}

void
trigger_sd_set_trigger_message (LogDriver *s, const gchar *message)
{
  TriggerSourceDriver *self = (TriggerSourceDriver *)s;

  g_free (self->options.message);
  self->options.message = g_strdup (message);
}

LogDriver *
trigger_sd_new (void)
{
  TriggerSourceDriver *self = g_new0 (TriggerSourceDriver, 1);

  log_src_driver_init_instance ((LogSrcDriver *)&self->super);

  self->super.super.super.init = trigger_sd_init;
  self->super.super.super.deinit = trigger_sd_deinit;

  log_source_options_defaults (&self->source_options);

  return (LogDriver *)self;
}

After that we add a thread pointer to the TriggerSource struct:

typedef struct
{
  LogSource super;

  TriggerOptions *options;
  GThread * workerThread;
} TriggerSource;

Then we rewrite the method which put the messages in the queue:

static void
trigger_source_triggered (gpointer s)
{
  TriggerSource *self = (TriggerSource *) s;
  LogMessage *msg;
  LogPathOptions path_options = LOG_PATH_OPTIONS_INIT;

  //There were weird segmentation violations without this line.
  iv_init();

  while(1) {
    if (log_source_free_to_send (&self->super)) {

      msg = log_msg_new_internal (LOG_INFO, self->options->message);
      path_options.ack_needed = FALSE;

      log_msg_refcache_start_producer(msg);
      log_pipe_queue (&self->super.super, msg, &path_options);
      log_msg_refcache_stop( );

      sleep(self->options->trigger_freq);

    } 
  }
}

The log_msg_refcache_start_producer and log_msg_refcache_stop are currently magical things. They are needed if we create a message.

Now we only need to create and start the thread:

static LogSource *
trigger_source_new (TriggerSourceDriver *owner, LogSourceOptions *options)
{
  TriggerSource *self = g_new0 (TriggerSource, 1);

  log_source_init_instance (&self->super);
  log_source_set_options (&self->super, options, 0, SCS_TRIGGER,
                          owner->super.super.id, NULL, FALSE);

  self->options = &owner->options;

  self->super.super.init = trigger_source_init;
  self->super.super.deinit = trigger_source_deinit;

  self->workerThread = create_worker_thread(&trigger_source_triggered, (void*)self, FALSE, NULL);

  return &self->super;
}

The trigger-source now runs in a different thread.
Please note, that there is a problem with the timestamp of the message.
The bug is under investigation.

Hello World!

So, here we are.
This is my first blog entry related to the development of the AMQP source for the syslog-ng project.

First of all, I want to welcome you, dear reader, let me introduce myself.

My name is MÓZES Ádám István, and I am a Computer Science student at the Budapest University of Technology and Economics at the moment. I expect to graduate in 2015.

This is my first participation in the Google Summer of Code program, and I am quite excited about the whole project. As I mentioned earlier, the goal of the project is to develop an AMQP source for syslog-ng. More details can be found here.

Of course, the development can be followed on GitHub, too, as soon as I create the relevant repository, I will post it. By the way, here is a link to my GitHub profile.

Feel free to contact me, any thoughts/questions related to the project are welcomed.