基于UIC构建单点登录

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 1. 前言 在阅读open-falcon源码时,意外发现它有个UIC(用户中心), 这个项目的代码实现比较简单,唯一问题是界面有点丑 传送门: UIC github地址 公司正好有个前后端分离的项目需要做登录,因此我打算基于UIC来做单点登录 2. 实现 关于单点登录的基本概念请阅读参考资料1自行脑补 2.1 login过程 注①② 也可采用cookie的方案,本方案是将sig存储在浏览器的localstorage中,每次请求后端API时,附带在Header头中 注③ 一般可以采用302跳转的方案,但是UIC项目本身是通过JS实现的Redirect到callback URL 2.2 logout过程 logout过程需要对UIC进行改造 调用UIC的logout函数,需要执行以下动作 remove UIC session 告知其它已经登录的系统的执行logout(需要每个参与系统需要提供接口, 清除session) 参考资料 单点登录原理与简单实现 PS: 仔细想想微信公众号的Oauth也是单点登录的过程 请我喝瓶饮料

August 8, 2018 · 1 min

redigo提示connection pool exhausted

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 1. 引言 线上的某个服务(Golang开发)使用Redis作为消息队列,使用的redis库是garyburd/redigo, 这两天出现如下错误 connection pool exhausted 2. 产生原因 阅读源码pool.go 阅读get()即可 type Pool struct { // Dial()方法返回一个连接,从在需要创建连接到的时候调用 Dial func() (Conn, error) // TestOnBorrow()方法是一个可选项,该方法用来诊断一个连接的健康状态 TestOnBorrow func(c Conn, t time.Time) error // 最大空闲连接数 MaxIdle int // 一个pool所能分配的最大的连接数目 // 当设置成0的时候,该pool连接数没有限制 MaxActive int // 空闲连接超时时间,超过超时时间的空闲连接会被关闭。 // 如果设置成0,空闲连接将不会被关闭 // 应该设置一个比redis服务端超时时间更短的时间 IdleTimeout time.Duration // 如果Wait被设置成true,则Get()方法将会阻塞 Wait bool ... ... } 以上异常的原因是这样的,在garyburd/redigo中,得到连接的步骤如下 尝试从空闲列表中,获得一个可用连接;如果成功直接返回,失败则尝试步骤2 如果当前的Active 连接数 < MaxActive,则尝试创建一个新连接;如果成功直接返回,失败则尝试步骤3 判断Wait为true则等待,否则报出异常 // ErrPoolExhausted is returned from a pool connection method (Do, Send, // Receive, Flush, Err) when the maximum number of database connections in the // pool has been reached. var ErrPoolExhausted = errors.New(&quot;redigo: connection pool exhausted&quot;) 3. 解决方法 设置MaxActive,设MaxActive=0(表示无限大)或者足够大。 设置Wait=true,当程序执行get(),无法获得可用连接时,将会暂时阻塞。 4. 完整的初始化连接池的代码 func NewRedisPool(redisConf context.RedisConf) *redis.Pool { address := fmt.Sprintf(&quot;%v:%v&quot;, redisConf.Host, redisConf.Port) dbOption := redis.DialDatabase(redisConf.Db) pwOption := redis.DialPassword(redisConf.Password) // **重要** 设置读写超时 readTimeout := redis.DialReadTimeout(time.Second * time.Duration(redisConf.ConTimeout)) writeTimeout := redis.DialWriteTimeout(time.Second * time.Duration(redisConf.ConTimeout)) conTimeout := redis.DialConnectTimeout(time.Second * time.Duration(redisConf.ConTimeout)) redisPool := &amp;redis.Pool{ // 从配置文件获取maxidle以及maxactive,取不到则用后面的默认值 MaxIdle: redisConf.MaxIdleConn, MaxActive: redisConf.MaxActiveConn, // **重要** 如果空闲列表中没有可用的连接 // 且当前Active连接数 &lt; MaxActive // 则等待 Wait: true, IdleTimeout: time.Duration(redisConf.IdleTimeout) * time.Second, Dial: func() (redis.Conn, error) { c, err := redis.Dial(&quot;tcp&quot;, address, dbOption, pwOption, readTimeout, writeTimeout, conTimeout) if err != nil { return nil, err } return c, nil }, } return redisPool }

June 4, 2018 · 1 min

轻量级服务器micro_httpd剖析

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 安装&配置 安装请参考参考资料1和2 安装完成以后,可以使用man micro_httpd 来获得更详细的说明 执行流程 代码解析 micro_httpd /* micro_httpd - really small HTTP server ** ** Copyright � 1999,2005 by Jef Poskanzer <jef@mail.acme.com>. ** All rights reserved. ** ** Redistribution and use in source and binary forms, with or without ** modification, are permitted provided that the following conditions ** are met: ** 1. Redistributions of source code must retain the above copyright ** notice, this list of conditions and the following disclaimer. ** 2. Redistributions in binary form must reproduce the above copyright ** notice, this list of conditions and the following disclaimer in the ** documentation and/or other materials provided with the distribution. ** ** THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ** ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE ** IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ** ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE ** FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL ** DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS ** OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) ** HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT ** LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY ** OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF ** SUCH DAMAGE. */ /* ####################################### 1.此程序对输入输出使用的是stdin和stdout 可以猜测它需要依赖其它程序对它的输入输出进行重定向 2.次程序没有创建socket服务 因此需要其它程序为其创建socket服务 ###################################### */ #include <sys/types.h> #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> #include <dirent.h> #include <ctype.h> #include <time.h> #include <sys/stat.h> #define SERVER_NAME "micro_httpd" #define SERVER_URL "http://www.acme.com/software/micro_httpd/" #define PROTOCOL "HTTP/1.0" #define RFC1123FMT "%a, %d %b %Y %H:%M:%S GMT" /* Forwards. */ //输出文件相关属性的详细信息 static void file_details( char* dir, char* name ); //返回出错页面 static void send_error( int status, char* title, char* extra_header, char* text ); //设置reponse static void send_headers( int status, char* title, char* extra_header, char* mime_type, off_t length, time_t mod ); //通过文件名推断文件mime_type static char* get_mime_type( char* name ); //URL 解码 static void strdecode( char* to, char* from ); //十六进制转换为十进制数 static int hexit( char c ); //用于文件名编码 static void strencode( char* to, size_t tosize, const char* from ); int main( int argc, char** argv ) { char line[10000], method[10000], path[10000], protocol[10000], idx[20000], location[20000], command[20000]; char* file; size_t len; int ich; struct stat sb; FILE* fp; struct dirent **dl; int i, n; if ( argc != 2 ) send_error( 500, "Internal Error", (char*) 0, "Config error - no dir specified." ); //更改工作目录 if ( chdir( argv[1] ) < 0 ) send_error( 500, "Internal Error", (char*) 0, "Config error - couldn't chdir()." ); if ( fgets( line, sizeof(line), stdin ) == (char*) 0 ) send_error( 400, "Bad Request", (char*) 0, "No request found." ); //解析请求 形如 GET /abc/index.html HTTP/1.0 if ( sscanf( line, "%[^ ] %[^ ] %[^ ]", method, path, protocol ) != 3 ) send_error( 400, "Bad Request", (char*) 0, "Can't parse request." ); while ( fgets( line, sizeof(line), stdin ) != (char*) 0 ) { if ( strcmp( line, "\n" ) == 0 || strcmp( line, "\r\n" ) == 0 ) break; } if ( strcasecmp( method, "get" ) != 0 ) send_error( 501, "Not Implemented", (char*) 0, "That method is not implemented." ); if ( path[0] != '/' ) send_error( 400, "Bad Request", (char*) 0, "Bad filename." ); file = &(path[1]); strdecode( file, file ); if ( file[0] == '\0' ) file = "./"; len = strlen( file ); if ( file[0] == '/' || strcmp( file, ".." ) == 0 || strncmp( file, "../", 3 ) == 0 || strstr( file, "/../" ) != (char*) 0 || strcmp( &(file[len-3]), "/.." ) == 0 ) send_error( 400, "Bad Request", (char*) 0, "Illegal filename." ); if ( stat( file, &sb ) < 0 ) send_error( 404, "Not Found", (char*) 0, "File not found." ); //请求路径是文件夹吗 if ( S_ISDIR( sb.st_mode ) ) { if ( file[len-1] != '/' ) { (void) snprintf( location, sizeof(location), "Location: %s/", path ); send_error( 302, "Found", location, "Directories must end with a slash." ); } (void) snprintf( idx, sizeof(idx), "%sindex.html", file ); if ( stat( idx, &sb ) >= 0 ) { file = idx; goto do_file; } //显示请求文件夹下的文件列表 send_headers( 200, "Ok", (char*) 0, "text/html", -1, sb.st_mtime ); (void) printf( "<html><head><title>Index of %s</title></head>\n<body bgcolor=\"#99cc99\"><h4>Index of %s</h4>\n<pre>\n", file, file ); n = scandir( file, &dl, NULL, alphasort ); if ( n < 0 ) perror( "scandir" ); else for ( i = 0; i < n; ++i ) file_details( file, dl[i]->d_name ); (void) printf( "</pre>\n<hr>\n<address><a href=\"%s\">%s</a></address>\n</body></html>\n", SERVER_URL, SERVER_NAME ); } else { //返回请求的文件 do_file: fp = fopen( file, "r" ); if ( fp == (FILE*) 0 ) send_error( 403, "Forbidden", (char*) 0, "File is protected." ); send_headers( 200, "Ok", (char*) 0, get_mime_type( file ), sb.st_size, sb.st_mtime ); while ( ( ich = getc( fp ) ) != EOF ) putchar( ich ); } (void) fflush( stdout ); exit( 0 ); } //输出文件相关属性的详细信息 static void file_details( char* dir, char* name ) { static char encoded_name[1000]; static char path[2000]; struct stat sb; char timestr[16]; strencode( encoded_name, sizeof(encoded_name), name ); (void) snprintf( path, sizeof(path), "%s/%s", dir, name ); if ( lstat( path, &sb ) < 0 ) (void) printf( "<a href=\"%s\">%-32.32s</a> ???\n", encoded_name, name ); else { (void) strftime( timestr, sizeof(timestr), "%d%b%Y %H:%M", localtime( &sb.st_mtime ) ); (void) printf( "<a href=\"%s\">%-32.32s</a> %15s %14lld\n", encoded_name, name, timestr, (int64_t) sb.st_size ); } } //返回出错页面 static void send_error( int status, char* title, char* extra_header, char* text ) { send_headers( status, title, extra_header, "text/html", -1, -1 ); (void) printf( "<html><head><title>%d %s</title></head>\n<body bgcolor=\"#cc9999\"><h4>%d %s</h4>\n", status, title, status, title ); (void) printf( "%s\n", text ); (void) printf( "<hr>\n<address><a href=\"%s\">%s</a></address>\n</body></html>\n", SERVER_URL, SERVER_NAME ); (void) fflush( stdout ); exit( 1 ); } //设置reponse header //状态 标题 mime_type 文件最后被修改的时间 static void send_headers( int status, char* title, char* extra_header, char* mime_type, off_t length, time_t mod ) { time_t now; char timebuf[100]; //\015\012 回车换行 (void) printf( "%s %d %s\015\012", PROTOCOL, status, title ); (void) printf( "Server: %s\015\012", SERVER_NAME ); now = time( (time_t*) 0 ); //将时间格式化后存在timebuf中 (void) strftime( timebuf, sizeof(timebuf), RFC1123FMT, gmtime( &now ) ); (void) printf( "Date: %s\015\012", timebuf ); if ( extra_header != (char*) 0 ) (void) printf( "%s\015\012", extra_header ); if ( mime_type != (char*) 0 ) (void) printf( "Content-Type: %s\015\012", mime_type ); if ( length >= 0 ) (void) printf( "Content-Length: %lld\015\012", (int64_t) length ); if ( mod != (time_t) -1 ) { (void) strftime( timebuf, sizeof(timebuf), RFC1123FMT, gmtime( &mod ) ); (void) printf( "Last-Modified: %s\015\012", timebuf ); } (void) printf( "Connection: close\015\012" ); (void) printf( "\015\012" ); } //通过文件名推断文件mime_type static char* get_mime_type( char* name ) { char* dot; dot = strrchr( name, '.' ); if ( dot == (char*) 0 ) return "text/plain; charset=iso-8859-1"; if ( strcmp( dot, ".html" ) == 0 || strcmp( dot, ".htm" ) == 0 ) return "text/html; charset=iso-8859-1"; if ( strcmp( dot, ".jpg" ) == 0 || strcmp( dot, ".jpeg" ) == 0 ) return "image/jpeg"; if ( strcmp( dot, ".gif" ) == 0 ) return "image/gif"; if ( strcmp( dot, ".png" ) == 0 ) return "image/png"; if ( strcmp( dot, ".css" ) == 0 ) return "text/css"; if ( strcmp( dot, ".au" ) == 0 ) return "audio/basic"; if ( strcmp( dot, ".wav" ) == 0 ) return "audio/wav"; if ( strcmp( dot, ".avi" ) == 0 ) return "video/x-msvideo"; if ( strcmp( dot, ".mov" ) == 0 || strcmp( dot, ".qt" ) == 0 ) return "video/quicktime"; if ( strcmp( dot, ".mpeg" ) == 0 || strcmp( dot, ".mpe" ) == 0 ) return "video/mpeg"; if ( strcmp( dot, ".vrml" ) == 0 || strcmp( dot, ".wrl" ) == 0 ) return "model/vrml"; if ( strcmp( dot, ".midi" ) == 0 || strcmp( dot, ".mid" ) == 0 ) return "audio/midi"; if ( strcmp( dot, ".mp3" ) == 0 ) return "audio/mpeg"; if ( strcmp( dot, ".ogg" ) == 0 ) return "application/ogg"; if ( strcmp( dot, ".pac" ) == 0 ) return "application/x-ns-proxy-autoconfig"; return "text/plain; charset=iso-8859-1"; } //用于URL 解码 static void strdecode( char* to, char* from ) { for ( ; *from != '\0'; ++to, ++from ) { if ( from[0] == '%' && isxdigit( from[1] ) && isxdigit( from[2] ) ) { *to = hexit( from[1] ) * 16 + hexit( from[2] ); from += 2; } else *to = *from; } *to = '\0'; } //十六进制转换为十进制数 static int hexit( char c ) { if ( c >= '0' && c <= '9' ) return c - '0'; if ( c >= 'a' && c <= 'f' ) return c - 'a' + 10; if ( c >= 'A' && c <= 'F' ) return c - 'A' + 10; return 0; /* shouldn't happen, we're guarded by isxdigit() */ } //用于文件名编码 static void strencode( char* to, size_t tosize, const char* from ) { int tolen; for ( tolen = 0; *from != '\0' && tolen + 4 < tosize; ++from ) { //ASCII 字符、数字 或者 "/_.-~" 中的一个 if ( isalnum(*from) || strchr( "/_.-~", *from ) != (char*) 0 ) { *to = *from; ++to; ++tolen; } else //对特殊字符进行编码 问题:什么样的特殊字符需要这样编码? { (void) sprintf( to, "%%%02x", (int) *from & 0xff ); to += 3; tolen += 3; } } *to = '\0'; } 参考资料 micro_http 下载地址 micro_httpd的安装与配置 后记 这是我2012年的文章,原来发表在ITeye,现在一并迁移过来 ...

January 22, 2018 · 8 min

我的监控世界观(1)

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 监控系统设计的目的是什么? 从哲学上,人所有的活动就是认识世界和改造世界。 IT意义上监控系统的目的是为了我们全面的了解我们的机器,系统的运行状态,以作出相应的判断和动作。它其实是更多的是侧重于认识世界。 监控系统设计的核心点是什么? 采集状态 状态判断 报警(或者触发某种动作) 图表展示(报表) 反馈 从经验来看,这4条的实现的难易程度,由难到易刚好是5 ->1,2 -> 3,4 监控系统的分类: 按紧急程度而言可分为2类 周期性监控 事件监控 (事件报警) 顾名思义,周期性监控一般是1分钟或5分钟来采集系统的状态,以cacti和nagios为代表 而事件监控一般是由被监控主体自主上报的某种事件,以sentry为代表(nagios的NRPE中也有trap) 它们适用于不同的场景,没有优劣之分 如果按照功能划分,又可分为: 安全监控 主机监控 应用监控 网络监控等等 有趣的规律: 以主机监控举例,早期的不少公司都是使用开源的监控软件作为工具,比如:cacati,nagios, ganglia,当物理机数量达到800~1000台左右时(我推测的),都逐步使用自己的开发的监控系统替换开源的监控软件 后记: 这篇文章其实是我2014年的文章,其实目前最火的应该是zabbix和open-falcon,其中我比较看好的是open-falcon,它在设计理念相较以前的监控系统是有一定很多创新和突破,且性能更好。

January 6, 2018 · 1 min

happybase put()操作默认使用批量?

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 起因:前段时间,我们把通过happybase向hbase 写数据的操作put() 操作换成了batch() 结果发现性能并没有提升 阅读代码,我发现put() 实现使用的就是批量插入 table.py def put(self, row, data, timestamp=None, wal=True): """Store data in the table. This method stores the data in the `data` argument for the row specified by `row`. The `data` argument is dictionary that maps columns to values. Column names must include a family and qualifier part, e.g. `cf:col`, though the qualifier part may be the empty string, e.g. `cf:`. Note that, in many situations, :py:meth:`batch()` is a more appropriate method to manipulate data. .. versionadded:: 0.7 `wal` argument :param str row: the row key :param dict data: the data to store :param int timestamp: timestamp (optional) :param wal bool: whether to write to the WAL (optional) """ with self.batch(timestamp=timestamp, wal=wal) as batch: batch.put(row, data) # 很明显是批量操作 batch.py ...

January 1, 2018 · 2 min

利用redis实现分布式环境下的限频

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc redis 本身有计数器,并且可以做原子的增1操作,特别适合用来做分布式环境下的限频 # coding:utf-8 import time import threading from redis import StrictRedis class Counter(object): def __init__(self, redis_url): self.redis_client = StrictRedis.from_url(redis_url) def increment(self, key): t = int(time.time()) sign = t / 60 redis_key = key + &#039;:&#039; + str(sign) counter = self.redis_client.incr(redis_key) # 注:设置key的失效时间没有必要和原子增1操作包含在一个事务中。 self.redis_client.expire(redis_key, 300) # 设置key的失效时间300 seconds return counter if __name__ == &#039;__main__&#039;: redis_url = &#039;redis://127.0.0.1:6379/0&#039; c = Counter(redis_url) for i in range(100): time.sleep(0.2) x = c.increment(&#039;hello&#039;) if x &gt; 50: print &quot;over limit&quot; print x 这里限频有个前提条件,就是分布式环境中时钟,必须尽量对齐。 在上面的例子中频率限制就是50次/分钟

January 1, 2018 · 1 min

时间片计算

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc ###起因 要进行预约时间的计算,干脆做一个简单的模块专门用于时间片的计算,等时间有空了会传到pypi上 1. 时间片的合并 ll = [] dd = { &quot;start_time&quot;: &quot;20:00&quot;, &quot;end_time&quot;: &quot;22:00&quot; } ll.append(dd) dd = { &quot;start_time&quot;: &quot;10:00&quot;, &quot;end_time&quot;: &quot;12:00&quot; } ll.append(dd) dd = { &quot;start_time&quot;: &quot;14:00&quot;, &quot;end_time&quot;: &quot;18:00&quot; } ll.append(dd) dd = { &quot;start_time&quot;: &quot;15:00&quot;, &quot;end_time&quot;: &quot;17:00&quot; } ll.append(dd) dd = { &quot;start_time&quot;: &quot;11:00&quot;, &quot;end_time&quot;: &quot;15:00&quot; } ll.append(dd) dd = { &quot;start_time&quot;: &quot;18:00&quot;, &quot;end_time&quot;: &quot;19:00&quot; } ll.append(dd) res = TimeFrame.merge(ll) record = res[0] assert record[&#039;start_time&#039;] == &quot;10:00&quot; assert record[&#039;end_time&#039;] == &quot;19:00&quot; record = res[1] assert record[&#039;start_time&#039;] == &quot;20:00&quot; assert record[&#039;end_time&#039;] == &quot;22:00&quot; 2. 时间片相减 item_list1 = [] dd = { &quot;start_time&quot;: &quot;08:00&quot;, &quot;end_time&quot;: &quot;15:00&quot;, } item_list1.append(dd) item_list2 = [] dd = { &quot;start_time&quot;: &quot;09:00&quot;, &quot;end_time&quot;: &quot;10:00&quot;, } item_list2.append(dd) dd = { &quot;start_time&quot;: &quot;11:00&quot;, &quot;end_time&quot;: &quot;12:00&quot;, } item_list2.append(dd) res_list = TimeFrame.minus(item_list1, item_list2) record = res_list[0] assert record[&#039;start_time&#039;] == &quot;08:00&quot; assert record[&#039;end_time&#039;] == &quot;09:00&quot; record = res_list[1] assert record[&#039;start_time&#039;] == &quot;10:00&quot; assert record[&#039;end_time&#039;] == &quot;11:00&quot; record = res_list[2] assert record[&#039;start_time&#039;] == &quot;12:00&quot; assert record[&#039;end_time&#039;] == &quot;15:00&quot; 代码如下 time_frame.py ...

January 1, 2018 · 2 min

指定出口IP,发起HTTP请求

# -*- coding=utf-8 -*- import socket import requests true_socket = socket.socket ipbind=&#039;xxx.xxx.xxx.xxx&#039; def bound_socket(*a, **k): sock = true_socket(*a, **k) # 端口是0, 操作系统会自动选择可用的端口号 sock.bind((ipbind, 0)) return sock socket.socket = bound_socket r = requests.get(&#039;http://ip.cn/&#039;) print r.status_code

January 1, 2018 · 1 min

用grequests实现并发http请求

起因 要用http请求探测服务的有效性,多进程,多线程,感觉似乎没有必要,看看有没有协程的方案 1. 简单用法 grequests 利用 requests和gevent库,做了一个简单封装,使用起来非常方便 import grequests import time import requests urls = [ &#039;https://docs.python.org/2.7/library/index.html&#039;, &#039;https://docs.python.org/2.7/library/dl.html&#039;, &#039;http://www.iciba.com/partial&#039;, &#039;http://2489843.blog.51cto.com/2479843/1407808&#039;, &#039;http://blog.csdn.net/woshiaotian/article/details/61027814&#039;, &#039;https://docs.python.org/2.7/library/unix.html&#039;, &#039;http://2489843.blog.51cto.com/2479843/1386820&#039;, &#039;http://www.bazhuayu.com/tutorial/extract_loop_url.aspx?t=0&#039;, ] def method1(): t1 = time.time() for url in urls: res = requests.get(url) #print res.status_code t2 = time.time() print &#039;method1&#039;, t2 - t1 def method2(): tasks = [grequests.get(u) for u in urls] t1 = time.time() res = grequests.map(tasks, size=3) # print res t2 = time.time() print &#039;method2&#039;, t2 - t1 def method3(): tasks = [grequests.get(u) for u in urls] t1 = time.time() res = grequests.map(tasks, size=6) # print res t2 = time.time() if __name__ == &#039;__main__&#039;: method1() method2() method3() 运行结果如下: ...

January 1, 2018 · 1 min

wget bind ip 失败

版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 起因: 我们的程序需要从文件服务器拉去文件(跨机房), 程序运行的机器有电信、联通、移动3线,3个IP 假定为 01_DX_IP, 01_LT_IP, 01_YD_IP 文件服务器也是电信、联通、移动3线,3个IP 02_DX_IP, 02_LT_IP, 02_YD_IP 其域名为myfile.com 使用wget拉去文件,为了保证出口带宽用的尽量上,绑定出口IP wget --bind-address=01_YD_IP -t 3 -T 120 --limit-rate=3M -S -O 1407856770354_858932.mp4 http://myfile.com/data10/sony/303/2014-08/12/1407856770354_858932.mp4 按照预先的想法,绑定了移动IP,正常DNS服务器会返回一个文件服务器的移动IP,速度应该不会太慢,事情情况下,下载速度不到100k 1. 排查: 1.1 观察文件下载连接 观察wget打印的日志,显示实际建立连接访问的是联通的IP Resolving myfile.com... 02_LT_IP, ... Connecting to myfile.com|02_LT_IP|:80... connected. HTTP request sent, awaiting response... ^C 使用lsof观察进程的连接情况 [xxx@c3n~]$ sudo lsof -p 133396 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME wget 133396 root cwd DIR 8,2 4096 59640871425 /datapool/log/inject/xxx.log wget 133396 root 6w REG 0,17 164815000 59756490241 /datapool/injectBase/1408203875738_777462.mp4 wget 133396 root 7u IPv4 1479434406 0t0 TCP 01_YD_IP:29657->02_LT_IP:http (ESTABLISHED) 注意这一行 TCP 01_YD_IP:29657->02_LT_IP:http (ESTABLISHED) 本地的bind address确实已经生效,已经绑定了移动IP, 但是文件服务器的IP地址却是联通IP, 那只能是DNS解析有问题了 ...

January 1, 2018 · 2 min