近期评论

为什么nginx+php-fpm组合老是报错504

最近碰到这个问题大为干火。

一会行,一会不行。大部分情况行,不行的情况还是比较多。

老是报错如下:

2015/04/06 19:48:02 [error] 9386#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 219.134.166.25, server: www.whyalive.org, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.whyalive.org"

数据库可以正常访问。

网上有一种说法就是php-fpm的IO模块是阻塞的,一旦php-fpm卡住就会出现上面那报错。

到底是不是呢?

首先来看下nginx的debug日志的关键部分:

2015/04/09 21:48:19 [debug] 26490#0: *59 fastcgi param: "HTTP_ACCEPT_ENCODING: gzip, deflate, sdch"
2015/04/09 21:48:19 [debug] 26490#0: *59 fastcgi param: "HTTP_ACCEPT_LANGUAGE: zh-CN,zh;q=0.8"
2015/04/09 21:48:19 [debug] 26490#0: *59 fastcgi param: "HTTP_COOKIE: comment_author_9bc7ae7025f96b8d32f1659c9394702e=%E9%A3%9E%E9%9B%AA; comment_author_email_9bc7ae7025f96b8d32f1659c9394702e=liufei2014%40souhu.com; wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_9bc7ae7025f96b8d32f1659c9394702e=geek%7C1428758595%7CDvDFWuxfykwmBUudazEaWQqgB8ljS1h5oZPyki1lFUU%7C829fbd4a44d854f8640e7db920598ea4a92c2fa1e5423900b4a85d8ea5f73555"
2015/04/09 21:48:19 [debug] 26490#0: *59 http cleanup add: 00000000012CD630
2015/04/09 21:48:19 [debug] 26490#0: *59 get rr peer, try: 1
2015/04/09 21:48:19 [debug] 26490#0: *59 socket 31
2015/04/09 21:48:19 [debug] 26490#0: *59 epoll add connection: fd:31 ev:80002005
2015/04/09 21:48:19 [debug] 26490#0: *59 connect to 127.0.0.1:9000, fd:31 #60
2015/04/09 21:48:19 [debug] 26490#0: *59 http upstream connect: -2
2015/04/09 21:48:19 [debug] 26490#0: *59 posix_memalign: 00000000012F3F00:128 @16
2015/04/09 21:48:19 [debug] 26490#0: *59 event timer add: 31: 30000:1428587329428
2015/04/09 21:48:19 [debug] 26490#0: *59 http finalize request: -4, "/index.php?p=281" a:1, c:3
2015/04/09 21:48:19 [debug] 26490#0: *59 http request count:3 blk:0
2015/04/09 21:48:19 [debug] 26490#0: *59 http finalize request: -4, "/index.php?p=281" a:1, c:2
2015/04/09 21:48:19 [debug] 26490#0: *59 http request count:2 blk:0
2015/04/09 21:48:19 [debug] 26490#0: *59 http run request: "/index.php?p=281"
2015/04/09 21:48:19 [debug] 26490#0: *59 http upstream check client, write event:1, "/index.php"
2015/04/09 21:48:19 [debug] 26490#0: *59 http upstream recv(): -1 (11: Resource temporarily unavailable)

 

2015/04/09 21:48:49 [debug] 26490#0: *59 event timer del: 31: 1428587329428
2015/04/09 21:48:49 [debug] 26490#0: *59 http upstream request: "/index.php?p=281"
2015/04/09 21:48:49 [debug] 26490#0: *59 http upstream send request handler
2015/04/09 21:48:49 [debug] 26490#0: *59 http next upstream, 4
2015/04/09 21:48:49 [debug] 26490#0: *59 free rr peer 1 4
2015/04/09 21:48:49 [debug] 26490#0: *59 http upstream send request handler
2015/04/09 21:48:49 [debug] 26490#0: *59 http next upstream, 4
2015/04/09 21:48:49 [debug] 26490#0: *59 free rr peer 1 4
2015/04/09 21:48:49 [error] 26490#0: *59 upstream timed out (110: Connection timed out) while connecting to upstream, client: 183.12.199.17, server: www.whyalive.org, request: "GET /?p=281 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.whyalive.org", referrer: "http://www.geektcp.com/?p=69"
2015/04/09 21:48:49 [debug] 26490#0: *59 finalize http upstream request: 504
2015/04/09 21:48:49 [debug] 26490#0: *59 finalize http fastcgi request
2015/04/09 21:48:49 [debug] 26490#0: *59 close http upstream connection: 31
2015/04/09 21:48:49 [debug] 26490#0: *59 finalize http fastcgi request
2015/04/09 21:48:49 [debug] 26490#0: *59 close http upstream connection: 31
2015/04/09 21:48:49 [debug] 26490#0: *59 free: 00000000012F3F00, unused: 48
2015/04/09 21:48:49 [debug] 26490#0: *59 reusable connection: 0
2015/04/09 21:48:49 [debug] 26490#0: *59 reusable connection: 0
2015/04/09 21:48:49 [debug] 26490#0: *59 http finalize request: 504, "/index.php?p=281" a:1, c:1
2015/04/09 21:48:49 [debug] 26490#0: *59 http special response: 504, "/index.php?p=281"
2015/04/09 21:48:49 [debug] 26490#0: *59 internal redirect: "/504.html?"
2015/04/09 21:48:49 [debug] 26490#0: *59 rewrite phase: 1
2015/04/09 21:48:49 [debug] 26490#0: *59 test location: "/"
2015/04/09 21:48:49 [debug] 26490#0: *59 test location: "/"

上面那部分其实用到了ngx_http_upstream_check_broken_connection这个函数,这个函数在调用recv函数时候不成功,该函数返回结果是-1,正因为recv不成功,导致后面重试调用ngx_http_upstream_next返回504错误。

首先要明确一个概念:客户端是nginx,服务端是nginx里面的upstream参数关联的php-fpm地址。

通过分析nginx源代码,发现504错误的原因有3个,分别是:

1、nginx设置的fastcgi缓存太小会导致这个问题,所以网上有一大堆的配置。

2、客户端nginx跟php-fpm建立socket连接时,数据写入失败,nginx会调用ngx_http_upstream_next抛出504错误。

3、客户端nginx读取php-fpm返回过来的数据失败,nginx会调用ngx_http_upstream_next抛出504错误

 

而我的服务器为什么很容易导致504错误,直接原因是第三种,nginx通过recv读取php-fpm返回过来的数据时超时报错了。

继续深入下去,这个问题又分为两种:

1、php-fpm已经准备好数据,并发送成功。那么这种情况应该是nginx程序代码的问题。

2、php-fpm卡住了,没有发送数据给nginx,那么nginx就会抛出504错误。

 

显然第一种的可能性极小,我觉得这个问题要追踪到php-fpm了。

极有可能是php-fpm卡住了,没有返回数据给nginx导致。问题是php-fpm的配置已经调了很多遍了还是这个问题。

目前的解决方案是不用socket连接。如果还不行,就要看看php-fpm的代码和参数了。

###################################

经过一天的测试和观察,再也没有504错误了,即便偶尔会卡住零点几秒,之前在这个时候会报504错误,但是现在没有出现这种情况了。这个问题终于解决。very nice!

 

下面是这两个函数的源码:

static void
ngx_http_upstream_check_broken_connection(ngx_http_request_t *r,
    ngx_event_t *ev)
{
    int                  n;
    char                 buf[1];
    ngx_err_t            err;
    ngx_int_t            event;
    ngx_connection_t     *c;
    ngx_http_upstream_t  *u;



    ngx_log_debug2(NGX_LOG_DEBUG_HTTP, ev->log, 0,
                   "http upstream check client, write event:%d, \"%V\"",
                   ev->write, &r->uri);

    c = r->connection;
    u = r->upstream;

    if (c->error) {
        if ((ngx_event_flags & NGX_USE_LEVEL_EVENT) && ev->active) {

            event = ev->write ? NGX_WRITE_EVENT : NGX_READ_EVENT;

            if (ngx_del_event(ev, event, 0) != NGX_OK) {
                ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
                return;
            }
        }

        if (!u->cacheable) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_CLIENT_CLOSED_REQUEST);
        }

        return;
    }

#if (NGX_HTTP_SPDY)
    if (r->spdy_stream) {
        return;
    }
#endif

#if (NGX_HAVE_KQUEUE)

    if (ngx_event_flags & NGX_USE_KQUEUE_EVENT) {

        if (!ev->pending_eof) {
            return;
        }

        ev->eof = 1;
        c->error = 1;

        if (ev->kq_errno) {
            ev->error = 1;
        }

        if (!u->cacheable && u->peer.connection) {
            ngx_log_error(NGX_LOG_INFO, ev->log, ev->kq_errno,
                          "kevent() reported that client prematurely closed "
                          "connection, so upstream connection is closed too");
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_CLIENT_CLOSED_REQUEST);
            return;
        }

        ngx_log_error(NGX_LOG_INFO, ev->log, ev->kq_errno,
                      "kevent() reported that client prematurely closed "
                      "connection");

        if (u->peer.connection == NULL) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_CLIENT_CLOSED_REQUEST);
        }

        return;
    }

#endif

#if (NGX_HAVE_EPOLLRDHUP)

    if ((ngx_event_flags & NGX_USE_EPOLL_EVENT) && ev->pending_eof) {
        socklen_t  len;

        ev->eof = 1;
        c->error = 1;

        err = 0;
        len = sizeof(ngx_err_t);

        /*
         * BSDs and Linux return 0 and set a pending error in err
         * Solaris returns -1 and sets errno
         */

        if (getsockopt(c->fd, SOL_SOCKET, SO_ERROR, (void *) &err, &len)
            == -1)
        {
            err = ngx_socket_errno;
        }

        if (err) {
            ev->error = 1;
        }

        if (!u->cacheable && u->peer.connection) {
            ngx_log_error(NGX_LOG_INFO, ev->log, err,
                        "epoll_wait() reported that client prematurely closed "
                        "connection, so upstream connection is closed too");
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_CLIENT_CLOSED_REQUEST);
            return;
        }

        ngx_log_error(NGX_LOG_INFO, ev->log, err,
                      "epoll_wait() reported that client prematurely closed "
                      "connection");

        if (u->peer.connection == NULL) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_CLIENT_CLOSED_REQUEST);
        }

        return;
    }

#endif

    n = recv(c->fd, buf, 1, MSG_PEEK);

    err = ngx_socket_errno;

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, ev->log, err,
                   "http upstream recv(): %d", n);

    if (ev->write && (n >= 0 || err == NGX_EAGAIN)) {
        return;
    }

    if ((ngx_event_flags & NGX_USE_LEVEL_EVENT) && ev->active) {

        event = ev->write ? NGX_WRITE_EVENT : NGX_READ_EVENT;

        if (ngx_del_event(ev, event, 0) != NGX_OK) {
            ngx_http_upstream_finalize_request(r, u,
                                               NGX_HTTP_INTERNAL_SERVER_ERROR);
            return;
        }
    }

    if (n > 0) {
        return;
    }

    if (n == -1) {
        if (err == NGX_EAGAIN) {
            return;
        }

        ev->error = 1;

    } else { /* n == 0 */
        err = 0;
    }

    ev->eof = 1;
    c->error = 1;

    if (!u->cacheable && u->peer.connection) {
        ngx_log_error(NGX_LOG_INFO, ev->log, err,
                      "client prematurely closed connection, "
                      "so upstream connection is closed too");
        ngx_http_upstream_finalize_request(r, u,
                                           NGX_HTTP_CLIENT_CLOSED_REQUEST);
        return;
    }

    ngx_log_error(NGX_LOG_INFO, ev->log, err,
                  "client prematurely closed connection");

    if (u->peer.connection == NULL) {
        ngx_http_upstream_finalize_request(r, u,
                                           NGX_HTTP_CLIENT_CLOSED_REQUEST);
    }
}

#####################################################################################

static void
ngx_http_upstream_next(ngx_http_request_t *r, ngx_http_upstream_t *u,
    ngx_uint_t ft_type)
{
    ngx_msec_t  timeout;
    ngx_uint_t  status, state;

    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "http next upstream, %xi", ft_type);

    if (u->peer.sockaddr) {

        if (ft_type == NGX_HTTP_UPSTREAM_FT_HTTP_403
            || ft_type == NGX_HTTP_UPSTREAM_FT_HTTP_404)
        {
            state = NGX_PEER_NEXT;

        } else {
            state = NGX_PEER_FAILED;
        }

        u->peer.free(&u->peer, u->peer.data, state);
        u->peer.sockaddr = NULL;
    }

    if (ft_type == NGX_HTTP_UPSTREAM_FT_TIMEOUT) {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, NGX_ETIMEDOUT,
                      "upstream timed out");
    }

    if (u->peer.cached && ft_type == NGX_HTTP_UPSTREAM_FT_ERROR) {
        status = 0;

        /* TODO: inform balancer instead */

        u->peer.tries++;

    } else {
        switch (ft_type) {

        case NGX_HTTP_UPSTREAM_FT_TIMEOUT:
            status = NGX_HTTP_GATEWAY_TIME_OUT;
            break;

        case NGX_HTTP_UPSTREAM_FT_HTTP_500:
            status = NGX_HTTP_INTERNAL_SERVER_ERROR;
            break;

        case NGX_HTTP_UPSTREAM_FT_HTTP_403:
            status = NGX_HTTP_FORBIDDEN;
            break;

        case NGX_HTTP_UPSTREAM_FT_HTTP_404:
            status = NGX_HTTP_NOT_FOUND;
            break;

        /*
         * NGX_HTTP_UPSTREAM_FT_BUSY_LOCK and NGX_HTTP_UPSTREAM_FT_MAX_WAITING
         * never reach here
         */

        default:
            status = NGX_HTTP_BAD_GATEWAY;
        }
    }

    if (r->connection->error) {
        ngx_http_upstream_finalize_request(r, u,
                                           NGX_HTTP_CLIENT_CLOSED_REQUEST);
        return;
    }

    if (status) {
        u->state->status = status;
        timeout = u->conf->next_upstream_timeout;

        if (u->peer.tries == 0
            || !(u->conf->next_upstream & ft_type)
            || (timeout && ngx_current_msec - u->peer.start_time >= timeout))
        {
#if (NGX_HTTP_CACHE)

            if (u->cache_status == NGX_HTTP_CACHE_EXPIRED
                && (u->conf->cache_use_stale & ft_type))
            {
                ngx_int_t  rc;

                rc = u->reinit_request(r);

                if (rc == NGX_OK) {
                    u->cache_status = NGX_HTTP_CACHE_STALE;
                    rc = ngx_http_upstream_cache_send(r, u);
                }

                ngx_http_upstream_finalize_request(r, u, rc);
                return;
            }
#endif

            ngx_http_upstream_finalize_request(r, u, status);
            return;
        }
    }

    if (u->peer.connection) {
        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                       "close http upstream connection: %d",
                       u->peer.connection->fd);
#if (NGX_HTTP_SSL)

        if (u->peer.connection->ssl) {
            u->peer.connection->ssl->no_wait_shutdown = 1;
            u->peer.connection->ssl->no_send_shutdown = 1;

            (void) ngx_ssl_shutdown(u->peer.connection);
        }
#endif

        if (u->peer.connection->pool) {
            ngx_destroy_pool(u->peer.connection->pool);
        }

        ngx_close_connection(u->peer.connection);
        u->peer.connection = NULL;
    }

    ngx_http_upstream_connect(r, u);
}

192 Responses to “为什么nginx+php-fpm组合老是报错504”

  • Yep, they completely changed the design of it for fallout

  • computer说道:

    Summer in qassim is hot but in winter it usually rains and weatheris beautiful.

  • Very rapidly this website will be famous amid all blog viewers, due to it’s good articles or
    reviews

  • This article provides clear idea designed for the new viewers
    of blogging,that actually hoow to do blogging and site-building.

  • With havin so much written content do you ever run into any
    problems of plagorism or copyright violation? My blog has a lot of unique content
    I’ve either authored myself or outsourced but it appears
    a lot of it is popping it up all over the web without my permission. Do you
    know any ways to help prevent content from being ripped off?
    I’d really appreciate it.

  • Rogelio Hubka说道:

    stomatolog wrzesnia

  • Kum Airola说道:

    best facial cleanser

  • Hey I know this is off topic but I was curious if you knew of any widgets I could add to my web page that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new articles.

  • An interesting discussion is worth comment. I think that you should write more on this topic, it might not be a taboo subject but generally people are not enough to speak on such topics. To the next. Cheers

  • resume template说道:

    Hello there! This is my first comment here so I just wanted to give a quick shout out and tell you I really enjoy reading through your posts. Can you suggest any other blogs/websites/forums that go over the same topics? Many thanks!

  • Rocco Neuman说道:

    We are a group of volunteers and starting a new scheme in our community. Your website provided us with helpful information to paintings on. You have done a formidable activity and our entire group can be thankful to you.

  • Greetings from Florida! I’m bored at work so I decided to browse your site on my iphone during lunch break. I enjoy the info you present here and can’t wait to take a look when I get home. I’m shocked at how quick your blog loaded on my phone .. I’m not even using WIFI, just 3G .. Anyhow, amazing site!

  • Thanks. It’s a good post.

  • I like your post. Nice!

  • Bert Kutlu说道:

    Hi, Neat post. There’s a problem along with your website in web explorer, might check this? IE nonetheless is the market chief and a large element of other folks will miss your great writing because of this problem.

  • Thanks. It’s a good post.

  • Nice post. Thank you.

  • apply for pr说道:

    Good post! :)

  • Cornelius Clar说道:

    Thanks for the share. My momma used to say “Life is ten percent what happens to us and ninety percent how we respond to it”.

  • Therefore these diseases cause a significant impact in restroom life and tested to be threatening to his personal life. https://www.levitradosageus24.com/

  • pr application说道:

    Great post!

  • You can certainly see your skills in the work you write. The world hopes for more passionate writers like you who aren’t afraid to say how they believe. Always go after your heart.

  • Nice post. Keep it up.

  • apply for pr说道:

    This is a good post! I like it.

  • Nice post. Keep it up.

  • WONDERFUL Post.thanks for share..more wait .. …

  • Nice post. Thank you!

  • apply for pr说道:

    Nice post. Thank you!

  • pr application说道:

    Nice post. Thank you!

  • Very nice post!!

  • I love this post. Great.

  • pr application说道:

    I love this post. Great.

  • apply pr说道:

    I love this post. Great.

  • I love this post. Great.

  • apply pr说道:

    Thanks!

  • Very good. I love it!

  • Very good. I love it!

  • Charlesmuh说道:

    [url=http://buyzofran.click/]zofran 8 mg[/url] [url=http://moduretic.bid/]moduretic[/url] [url=http://buyavana.bid/]avana[/url] [url=http://cialis.tech/]cialis for daily use[/url]

  • Peteradjug说道:

    [url=http://vpxl.stream/]vpxl[/url] [url=http://levitra-price.party/]levitra price per pill[/url] [url=http://tamoxifen-9.top/]tamoxifen citrate nolvadex[/url] [url=http://buy-cipro.gdn/]buy cipro online canada[/url] [url=http://singulair-generic.party/]continue[/url] [url=http://albuterol-sulfate-inhalation-solution.bid/]albuterol[/url] [url=http://buyavodart.us/]buy avodart[/url] [url=http://buyatenolol.club/]buy atenolol[/url] [url=http://citalopram-20-mg.science/]citalopram[/url]

  • BennyMit说道:

    [url=http://buylipitor.site/]cost of lipitor[/url] [url=http://viagraforsale.date/]prescription for viagra[/url] [url=http://furosemide-lasix.party/]lasix furosemide[/url] [url=http://tretinoin-cream-0-05.us/]cheap tretinoin cream .025[/url] [url=http://retinaonline.us/]buy retin-a micro[/url] [url=http://proscar-online.bid/]proscar finasteride[/url]

  • MichaelTef说道:

    [url=http://mentat.top/]this site[/url] [url=http://viagraprice.link/]viagra price[/url] [url=http://buyprometriumonline.bid/]prometrium 100mg[/url] [url=http://crestoronline.site/]crestor[/url] [url=http://seroquelonline.site/]seroquel 100 mg[/url] [url=http://glucophage-xr.gdn/]glucophage xr[/url] [url=http://busparonline.review/]buspar online[/url] [url=http://buychloromycetin.party/]chloromycetin[/url]

  • Peteradjug说道:

    [url=http://orderdoxycycline.gdn/]doxycycline[/url] [url=http://fluoxetine-hcl.science/]fluoxetine[/url] [url=http://priceofcialis.click/]price of cialis[/url] [url=http://buyvardenafil.gdn/]vardenafil[/url] [url=http://buycitalopram5.top/]buy citalopram online[/url] [url=http://cafergot-2015.top/]related site[/url] [url=http://buyneurontin.gdn/]buy neurontin[/url]

  • BennyMit说道:

    [url=https://sexchat.us.com/]sex chat[/url]

  • EugeneCib说道:

    [url=http://valtrex2017.us.com/]valtrex[/url]

  • Tracyarepe说道:

    [url=http://buydoxycycline.us.com/]doxycycline no script needed[/url] [url=http://clindamycin365.us.com/]buy clindamycin[/url]

  • MichaelTef说道:

    [url=http://nexiumonline.site/]nexium[/url] [url=http://atenolol50mg.site/]more helpful hints[/url] [url=http://busparonline.review/]buspar[/url] [url=http://ashwagandha.bid/]ashwagandha brain fog[/url] [url=http://buyzetiaonline.gdn/]when will zetia go generic[/url] [url=http://chloromycetin.bid/]chloromycetin[/url] [url=http://dipyridamole.bid/]dipyridamole[/url] [url=http://mevacor.top/]generic for mevacor[/url]

  • Tracyarepe说道:

    [url=http://cefadroxil.webcam/]buy cefadroxil[/url] [url=http://buyviagraonline.us.com/]Viagra By Mail Order[/url] [url=http://lasixonline.site/]lasix[/url] [url=http://buynoroxin.accountant/]noroxin online[/url] [url=http://orlistat.bid/]disgrasil orlistat 120 mg[/url] [url=http://bupropionxl.date/]bupropion[/url] [url=http://levlen.top/]levlen without prescription[/url] [url=http://furosemide-6.top/]furosemide[/url]

  • Aarontex说道:

    wh0cd423818 [url=http://chloromycetin.cricket/]chloromycetin online[/url] [url=http://buyglucophage.site/]glucophage[/url] [url=http://doxycycline-hyclate.us/]100mg doxycycline[/url] [url=http://hairlosscream.bid/]hair loss cream[/url]

  • Charlesmuh说道:

    wh0cd729259 [url=http://genericlipitor.date/]lipitor[/url] [url=http://buymotilium.us.com/]where to buy motilium[/url] [url=http://advair-diskus-250-50.us/]generic for advair diskus[/url] [url=http://augmentin-antibiotic.cricket/]augmentin antibiotic[/url] [url=http://cialiscostperpill.eu/]cialis[/url] [url=http://clindamycinphosphate.date/]clindamycin topical[/url] [url=http://generic-wellbutrin.party/]wellbutrin 150 mg[/url]

  • BennyMit说道:

    wh0cd680910 [url=http://2010boycott.com/]orlistat 120mg[/url] [url=http://generic-propecia.cricket/]where to buy finasteride[/url] [url=http://elimitecream.site/]permethrin cost[/url] [url=http://amitriptylinehcl.us/]amitriptyline[/url] [url=http://buydoxycycline.info/]buy doxycycline[/url] [url=http://buyglucotrol.gdn/]buy glucotrol[/url] [url=http://prednisolone-5mg.science/]prednisolone 5 mg[/url] [url=http://isoniazid.bid/]isoniazid 300 mg[/url] [url=http://dilantin.us/]dilantin[/url]

  • BennyMit说道:

    wh0cd874318 [url=http://atarax365.us.com/]atarax online[/url] [url=http://caverta.gdn/]caverta without prescription[/url] [url=http://buy-prednisolone.us/]buy prednisolone[/url] [url=http://buyantabuse.info/]antabuse[/url] [url=http://arimidex-online.gdn/]arimidex[/url] [url=http://buyavalide.science/]buy avalide[/url] [url=http://buyzithromax8.top/]buy zithromax[/url]

  • EugeneCib说道:

    wh0cd856669 [url=http://bupropion.fashion/]bupropion sr 150[/url] [url=http://buyzithromax.club/]buying zithromax[/url] [url=http://generic-for-cymbalta.cricket/]cymbalta prices[/url] [url=http://cost-of-advair.science/]advair asthma[/url] [url=http://kamagra-online.bid/]kamagra online[/url] [url=http://silagra.gdn/]silagra online[/url]

  • AlfredSek说道:

    wh0cd485215 [url=http://furosemide-40-mg.bid/]furosemide iv[/url] [url=http://buyatenolol2013.top/]buy atenolol online[/url] [url=http://buynolvadex.us.com/]BUY NOLVADEX[/url] [url=http://buyprometriumonline.gdn/]prometrium cream[/url] [url=http://torsemide.us/]torsemide 20 mg tablet[/url] [url=http://benicarhct.site/]benicar hct[/url] [url=http://sildenafil-100mg.science/]sildenafil[/url]

  • Kennethrok说道:

    wh0cd452209 [url=http://wellbutrinxl.site/]wellbutrin mg[/url] [url=http://buyadvair.info/]advair[/url] [url=http://brandlevitra20mg.gdn/]example here[/url] [url=http://inderalonline.link/]inderal online[/url]

  • Aarontex说道:

    wh0cd773016 [url=http://proscar-online.us/]found it for you[/url] [url=http://sildenafil-100mg.us/]sildenafil 100mg[/url] [url=http://stratera.click/]atomoxetine strattera[/url] [url=http://methylprednisolone.click/]going here[/url] [url=http://buyphenergan.website/]buy phenergan[/url]

  • MichaelTef说道:

    wh0cd353214 [url=http://cymbalta60mg.link/]homepage here[/url] [url=http://singulaironline.review/]singulair online[/url] [url=http://diclofenac-50mg.in.net/]diclofenac[/url] [url=http://tenormin.christmas/]tenormin[/url] [url=http://cymbaltamedication.review/]example here[/url] [url=http://rimonabant-2.top/]rimonabant acomplia[/url]

  • MichaelTef说道:

    wh0cd119047 [url=http://tricor.club/]buy tricor[/url] [url=http://remeron.club/]remeron[/url] [url=http://zyrtec.club/]zyrtec[/url]

  • Aarontex说道:

    wh0cd766983 [url=http://valtrex2016.us.com/]Valtrex[/url] [url=http://motilium365.us.com/]buy motilium[/url]

  • BennyMit说道:

    wh0cd755363 [url=http://viagracoupons.review/]viagra coupons[/url] [url=http://phenergan.fashion/]phenergan[/url] [url=http://buy-methotrexate.website/]methotrexate[/url] [url=http://orderviagra.review/]buy viagra online mastercard[/url] [url=http://nolvadexforsale.review/]nolvadex tamoxifen[/url]

  • Charlesmuh说道:

    wh0cd966429 [url=http://bentylmusclerelaxer.us.com/]bentyl tablets[/url]

  • AlfredSek说道:

    wh0cd225811 [url=http://lopid.club/]lopid[/url] [url=http://naprosyn.club/]naprosyn[/url] [url=http://xalatan.club/]xalatan[/url] [url=http://nimotop.club/]nimotop[/url] [url=http://vytorin.club/]vytorin 10 40[/url]

  • Charlesmuh说道:

    wh0cd693960 [url=http://cafergot2017.us.com/]cafergot & internet pharmacy[/url]

  • MichaelTef说道:

    wh0cd505164 [url=http://caverta.party/]caverta[/url]

  • Kennethrok说道:

    wh0cd620695 [url=http://phenergan.us.com/]buy phenergan[/url]

  • Charlesmuh说道:

    wh0cd944663 [url=http://trazodone1.top/]bonuses[/url] [url=http://albendazole16.us/]cheap albendazole[/url] [url=http://buyarimidex500.us/]buy arimidex[/url]

  • JudithBor说道:

    wh0cd921918 [url=http://costofviagra2017.com/]VIAGRA PRESCRIPTION[/url]

  • JudithBor说道:

    wh0cd974827 [url=http://costofviagra2017.com/]viagra online[/url]

  • BennyMit说道:

    wh0cd393109 [url=http://albuterol2012.top/]albuterol[/url] [url=http://levaquin-365.gdn/]order levaquin[/url] [url=http://buytadacip15.top/]tadacip[/url] [url=http://zithromax2.top/]zithromax 500[/url] [url=http://buylisinopril2015.top/]lisinopril medication[/url] [url=http://stromectol7.top/]stromectol[/url]

  • Kennethrok说道:

    wh0cd535870 [url=http://allopurinol24.us.org/]allopurinol[/url] [url=http://citalopram20.us.com/]citalopram 20 mg[/url]

  • Charlesmuh说道:

    wh0cd810619 [url=http://yasmin365.us.com/]yasmin online[/url] [url=http://effexor.biz/]effexor[/url] [url=http://onlineelimite.us.com/]online elimite[/url] [url=http://albuterolnebulizer.us.com/]albuterol online[/url] [url=http://effexor365.us.com/]cheap effexor[/url]

  • Saved as a favorite, I like your website!

  • new invention说道:

    You are so interesting! I do not think I have read through a single thing like this before. So nice to find another person with a few genuine thoughts on this subject matter. Seriously.. thanks for starting this up. This web site is something that’s needed on the internet, someone with a little originality!

  • I am really pleased to read this weblog posts which consists of
    lots of valuable data, thanks for providing these kinds of information.

  • Hi there! I’m at work surfing around your blog from my new iphone 3gs!
    Just wanted to say I love reading your blog and look forward to all your posts!
    Carry on the excellent work!

  • I’ve been exploring for a little bit for any high quality articles or blog posts in this sort of area .
    Exploring in Yahoo I eventually stumbled
    upon this site. Reading this info So i’m happy to express that I have an incredibly excellent uncanny feeling I
    found out just what I needed. I so much undoubtedly will make
    sure to do not forget this website and give it a look on a continuing
    basis.

  • I love it whenever people come together and share views. Great site, keep it
    up!

  • ekalyan说道:

    Heya i am for the first time here. I found this board and I find
    It truly useful & it helped me out a lot. I hope
    to give something back and aid others like you aided me.

  • This page truly has all the information and facts I
    wanted concerning this subject and didn’t know who
    to ask.

  • Hello there, I do believe your blog could possibly be having browser compatibility problems.
    Whenever I look at your website in Safari, it looks fine however, when opening in Internet
    Explorer, it has some overlapping issues. I merely wanted
    to give you a quick heads up! Aside from that, excellent site!

  • Aw, this was a very good post. Taking a few minutes and actual effort to generate a really good article…
    but what can I say… I procrastinate a lot and never seem
    to get nearly anything done.

  • Good blog you have got here.. It’s hard to find excellent writing like yours nowadays.
    I truly appreciate people like you! Take care!!

  • You are so awesome! I do not think I’ve read a single thing like that before.
    So nice to find somebody with a few original thoughts on this topic.
    Seriously.. thank you for starting this up. This website is one thing
    that is needed on the web, someone with a little originality!

  • Wow that was unusual. I just wrote an very long comment but after I
    clicked submit my comment didn’t show up. Grrrr…
    well I’m not writing all that over again. Anyhow, just wanted to say great blog!

  • Hello there! I could have sworn I’ve been to this site before but
    after browsing through some of the post I realized it’s new to me.
    Anyways, I’m definitely glad I found it and I’ll be book-marking and checking back often!

  • RobertTek说道:

    [url=https://paydayloans.us.org/]cash advance[/url]

  • Thankfulness to my father who stated to me concerning this blog,
    this webpage is truly amazing.

  • When some one searches for his required thing,
    thus he/she needs to be available that in detail, thus
    that thing is maintained over here.

  • Hi there mates, how is the whole thing, and what you wish for to say concerning this post, in my view its actually amazing in favor of
    me.

  • Saved as a favorite, I really like your site!

  • What’s up, yup this article is in fact fastidious and I have learned lot of
    things from it regarding blogging. thanks.

  • Lyle Magg说道:

    As a Newbie, I am continuously exploring online for articles that can help me. Thank you

  • have a peek here说道:

    Hello there, just became alert to your blog through Google,
    and found that it is really informative. I’m gonna watch out for brussels.
    I’ll be grateful if you continue this in future.
    A lot of people will be benefited from your writing.
    Cheers!

  • this content说道:

    I am really loving the theme/design of your blog.
    Do you ever run into any web browser compatibility
    problems? A couple of my blog readers have complained about my blog not working correctly in Explorer
    but looks great in Opera. Do you have any ideas to help fix this problem?

  • go right here说道:

    Hi I am so thrilled I found your website, I really found you by mistake, while I was looking on Bing for
    something else, Nonetheless I am here now and would just like to say thank you for a fantastic post and a all round interesting blog (I also love
    the theme/design), I don’t have time to read through it all at
    the minute but I have book-marked it and also added in your RSS feeds, so when I have time I will be back to read a great
    deal more, Please do keep up the fantastic job.

  • Keep on working, great job!

  • Thanks , I have just been searching for info approximately this topic for ages
    and yours is the greatest I’ve found out till now. But,
    what in regards to the conclusion? Are you positive concerning the supply?

  • It’s a pity you don’t have a donate button! I’d certainly donate to this
    fantastic blog! I guess for now i’ll settle for book-marking and adding your RSS feed to my Google account.

    I look forward to brand new updates and will
    share this blog with my Facebook group. Talk
    soon!

  • Please let me know if you’re looking for a article author for
    your weblog. You have some really great articles and I feel
    I would be a good asset. If you ever want to take
    some of the load off, I’d absolutely love to write some content for
    your blog in exchange for a link back to mine. Please blast me an email
    if interested. Cheers!

  • I enjoy what you guys are up too. This sort of
    clever work and coverage! Keep up the terrific works guys I’ve added you guys
    to my personal blogroll.

  • Free followers说道:

    Hello to every body, it’s my first pay a visit of this website;
    this weblog carries amazing and really fine data designed for readers.

  • Opersonalized说道:

    Keep on writing, great job!

  • heart necklace说道:

    Awesome site you have here but I was wondering if you knew of any discussion boards that cover the same topics talked about in this article?
    I’d really like to be a part of online community where I
    can get feed-back from other experienced individuals that share the same interest.
    If you have any recommendations, please let me know. Thanks!

  • For latest information you have to pay a quick visit internet and on world-wide-web
    I found this web site as a best web site for latest updates.

  • Fabulous, what a weblog it is! This webpage gives helpful information to us, keep
    it up.

  • Thank you for any other magnificent article.
    Where else may just anyone get that kind of information in such a perfect method of writing?
    I’ve a presentation next week, and I am on the search for such
    info.

  • Hi there! I’m at work surfing around your blog from my new iphone 3gs!
    Just wanted to say I love reading through your blog and look forward to all your posts!
    Carry on the superb work!

  • invention patent说道:

    Amazing blog! Do you have any recommendations for aspiring writers?
    I’m hoping to start my own site soon but I’m
    a little lost on everything. Would you advise starting with a free platform like WordPress or
    go for a paid option? There are so many choices out there that I’m totally overwhelmed ..

    Any tips? Cheers!

  • Hi, I do believe this is an excellent blog. I stumbledupon it 😉 I will return once again since
    I bookmarked it. Money and freedom is the best way to change, may
    you be rich and continue to help others.

  • After I initially left a comment I appear to have clicked on the -Notify me when new comments are added- checkbox and now whenever a comment is added
    I get four emails with the same comment. Perhaps there is a way you can remove me
    from that service? Thank you!

  • I absolutely love your blog.. Pleasant colors & theme.
    Did you build this amazing site yourself? Please reply back as I’m hoping to create my
    own personal website and would love to know where you got
    this from or just what the theme is called. Thank you!

  • It is in reality a nice and useful piece of information. I’m happy that you shared
    this helpful information with us. Please stay us informed like this.
    Thank you for sharing.

  • Simon Arias说道:

    Great info. Lucky me I came across your website by accident (stumbleupon).
    I have bookmarked it for later!

  • Hi there, every time i used to check webpage posts here early in the daylight, since i love to gain knowledge
    of more and more.

  • Simon Arias说道:

    Do you mind if I quote a few of your posts as long as I provide credit and sources
    back to your weblog? My blog site is in the exact same area of interest
    as yours and my users would truly benefit from a lot of the information you provide here.
    Please let me know if this okay with you.
    Regards!

  • Hello, I read your new stuff on a regular
    basis. Your story-telling style is awesome, keep it up!

  • Hello! I’ve been reading your blog for a long time now and finally got the courage to go ahead and
    give you a shout out from Humble Texas! Just wanted to mention keep up the excellent job!

  • I’ve been browsing on-line greater than three hours nowadays,
    yet I by no means discovered any attention-grabbing article like yours.
    It’s beautiful value sufficient for me. In my opinion, if all site owners and
    bloggers made good content as you probably did, the internet will be a lot more helpful
    than ever before.

  • new product idea说道:

    Hi there! I could have sworn I’ve visited this website before but after going through some of the articles I
    realized it’s new to me. Anyways, I’m certainly pleased
    I stumbled upon it and I’ll be bookmarking it and
    checking back regularly!

  • play school说道:

    Saved as a favorite, I really like your blog!

  • This paragraph is in fact a pleasant one it assists new web viewers, who are wishing for blogging.

  • Thanks for finally talking about > 为什么nginx+php-fpm组合老是报错504 < Loved it!

  • Does your site have a contact page? I’m having trouble locating it but, I’d
    like to shoot you an e-mail. I’ve got some recommendations for your blog you might
    be interested in hearing. Either way, great website and I look forward to seeing it expand over time.

  • Greetings from Colorado! I’m bored to death at work so I decided to browse your blog on my iphone during lunch break.

    I love the info you provide here and can’t wait
    to take a look when I get home. I’m shocked at how fast your blog loaded on my mobile ..
    I’m not even using WIFI, just 3G .. Anyhow, great site!

  • Oh my goodness! Amazing article dude! Thanks, However I am having issues with
    your RSS. I don’t know why I can’t subscribe to it.
    Is there anybody else getting identical RSS issues?

    Anyone that knows the answer can you kindly respond?

    Thanks!!

  • We absolutely love your blog and find a lot of your post’s to be
    precisely what I’m looking for. Does one
    offer guest writers to write content for yourself? I wouldn’t mind
    creating a post or elaborating on many of the subjects you write about here.
    Again, awesome blog!

  • I will immediately snatch your rss feed as I can’t find your e-mail subscription hyperlink
    or newsletter service. Do you’ve any? Kindly permit me recognise so
    that I could subscribe. Thanks.

  • invention ideas说道:

    I really like your blog.. very nice colors & theme. Did you
    make this website yourself or did you hire someone to do it for you?
    Plz respond as I’m looking to design my own blog and would like to know where u got
    this from. thanks

  • how to patent说道:

    Good respond in return of this difficulty with solid arguments and explaining everything concerning that.

  • It’s really a cool and helpful piece of info.
    I’m happy that you just shared this useful info with us.

    Please keep us up to date like this. Thank you for sharing.

  • Appreciating the persistence you put into your website and detailed information you present.

    It’s great to come across a blog every once in a while that isn’t the
    same old rehashed information. Great read! I’ve saved your site and I’m adding your
    RSS feeds to my Google account.

  • Appreciating the dedication you put into your website and in depth information you
    offer. It’s great to come across a blog every once in a
    while that isn’t the same out of date rehashed information.
    Excellent read! I’ve bookmarked your site and I’m including your RSS feeds to my Google account.

  • My brother suggested I might like this blog. He was totally right.
    This post truly made my day. You can not imagine simply how much time I had spent for this info!

    Thanks!

  • Hello there, just became aware of your blog through
    Google, and found that it is really informative. I’m gonna watch out for brussels.

    I will be grateful if you continue this in future. Numerous people
    will be benefited from your writing. Cheers!

  • Howdy would you mind letting me know which hosting
    company you’re working with? I’ve loaded your blog in 3 completely different web browsers
    and I must say this blog loads a lot quicker then most.

    Can you suggest a good internet hosting provider
    at a honest price? Kudos, I appreciate it!

  • I must thank you for the efforts you’ve put in writing this blog.
    I really hope to check out the same high-grade blog posts
    by you later on as well. In truth, your creative writing abilities has
    motivated me to get my own site now 😉

  • I do not know if it’s just me or if everybody else experiencing problems with your website.
    It appears as if some of the written text on your posts are running off the
    screen. Can someone else please provide feedback and let me know if this is happening to them as well?

    This could be a issue with my internet browser because I’ve had this happen before.
    Thank you

  • Quality articles or reviews is the crucial to invite the people to visit
    the site, that’s what this web page is providing.

  • whiteboard tape说道:

    I’ll right away clutch your rss as I can’t in finding your email subscription hyperlink or e-newsletter service.
    Do you’ve any? Please let me recognize so that I could subscribe.

    Thanks.

  • Spot on with this write-up, I actually believe this website
    needs a lot more attention. I’ll probably be returning to read through more, thanks for the info!

  • WOW just what I was searching for. Came here by searching for packaging

  • Sweet blog! I found it while searching on Yahoo News. Do you
    have any tips on how to get listed in Yahoo News?

    I’ve been trying for a while but I never seem to get there!
    Cheers

  • pallet returns说道:

    Fantastic site you have here but I was wondering if you knew of any community forums that cover the same topics talked about here?
    I’d really like to be a part of online community where I can get
    responses from other knowledgeable people that share the same
    interest. If you have any recommendations, please let me know.
    Thanks a lot!

  • book envelopes说道:

    Hey very nice blog!

  • Thanks on your marvelous posting! I genuinely
    enjoyed reading it, you happen to be a great author.I will be sure to
    bookmark your blog and may come back at some point.

    I want to encourage you continue your great writing, have a nice evening!

  • hook & loop tape说道:

    This excellent website truly has all the information I wanted concerning this subject and didn’t know who to ask.

  • recycling game说道:

    Hello, I enjoy reading all of your article post. I wanted to write a little comment
    to support you.

  • Spot on with this write-up, I honestly believe that this amazing
    site needs a lot more attention. I’ll probably be back again to read through more, thanks for the info!

  • I could not refrain from commenting. Perfectly written!

  • Hey there, You have done a fantastic job. I will certainly digg it and personally suggest to my friends.

    I’m sure they’ll be benefited from this website.

  • I just could not depart your site prior to suggesting that I actually loved the standard
    information an individual provide to your visitors? Is going to be back frequently in order to inspect new posts

  • Wow, fantastic blog layout! How long have you ever been blogging for?
    you make running a blog glance easy. The overall glance of your website is wonderful,
    let alone the content!

  • Hello there! This is my 1st comment here so I just wanted
    to give a quick shout out and tell you I
    genuinely enjoy reading through your posts. Can you recommend
    any other blogs/websites/forums that cover the same subjects?
    Thanks for your time!

  • Hi, I think your website might be having browser compatibility
    issues. When I look at your blog in Ie, it looks fine but when opening in Internet Explorer,
    it has some overlapping. I just wanted to give you a quick heads up!
    Other then that, great blog!

  • There is definately a lot to find out about this topic.
    I love all of the points you have made.

  • Wow, fantastic blog layout! How long have you been blogging for?
    you made blogging look easy. The overall look of
    your website is excellent, let alone the content!

  • karaj upvc说道:

    This design is incredible! You obviously know how to keep a
    reader amused. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Excellent job.
    I really loved what you had to say, and more than that, how you
    presented it. Too cool!

  • karaj upvc说道:

    Hello, I enjoy reading through your post. I wanted to write a little comment
    to support you.

  • karaj upvc说道:

    Yes! Finally something about karaj upvc.

  • karaj upvc说道:

    I could not refrain from commenting. Exceptionally well written!

  • karaj说道:

    Hi there, all is going fine here and ofcourse every one is sharing facts, that’s really good, keep up writing.

  • karaj upvc说道:

    Yes! Finally someone writes about karaj.

  • I’ve been browsing on-line greater than three hours nowadays, yet I by no means found any
    interesting article like yours. It’s lovely value enough for me.
    In my view, if all webmasters and bloggers made good content as you did, the internet will likely be much
    more useful than ever before.

  • karaj upvc说道:

    Hey would you mind letting me know which web host you’re utilizing?

    I’ve loaded your blog in 3 completely different internet browsers and I must say this blog loads
    a lot quicker then most. Can you suggest a good web hosting provider at
    a reasonable price? Cheers, I appreciate it!

  • Very rapidly this web page will be famous amid all blogging and site-building
    visitors, due to it’s good articles

  • Heya i’m for the first time here. I came across this board and I find It truly helpful & it helped me out much.
    I am hoping to present something back and aid others
    such as you helped me.

  • Unquestionably believe that which you stated. Your favorite reason seemed to be on the web the simplest thing to be aware of.

    I say to you, I definitely get irked while people think about
    worries that they just do not know about. You managed to hit
    the nail upon the top as well as defined out the whole thing without having side effect , people could take
    a signal. Will probably be back to get more. Thanks

  • Thanks for your personal marvelous posting! I genuinely enjoyed reading it, you will be
    a great author.I will be sure to bookmark your blog and
    will come back someday. I want to encourage you continue your great
    work, have a nice holiday weekend!

  • Sun tattoo说道:

    Hello! I know this is kinda off topic but I was wondering which blog platform are you using for
    this site? I’m getting sick and tired of WordPress because I’ve had problems with hackers and I’m looking at alternatives for another platform.
    I would be great if you could point me in the direction of a
    good platform. Symbol tattoo

  • Hi there! This post could not be written any better! Reading through this post reminds me of my old room mate! He always kept chatting about this. I will forward this post to him. Fairly certain he will have a good read. Thanks for sharing!

  • F*ckin?tremendous things here. I am very glad to see your post. Thanks a lot and i’m looking forward to contact you. Will you please drop me a e-mail?

  • Hey would you mind letting me know which webhost you’re working with?

    I’ve loaded your blog in 3 completely different internet browsers and
    I must say this blog loads a lot faster then most.
    Can you recommend a good hosting provider at a honest price?
    Kudos, I appreciate it!

  • domino win说道:

    Hello mates, good piece of writing and pleasant urging commented at this place, I am truly enjoying
    by these.

  • Grocery Delivery说道:

    This is the most toally rad site I’ve ever seen!

  • Fake Shoes说道:

    Try not to laugh!

  • Office Picture说道:

    Have yourself a very good day!

Leave a Reply