Golang 内存分配优化
版权声明 本站原创文章 由 萌叔 发表 转载请注明 萌叔 | http://vearne.cc 1. 前言 如果你正在使用imroc/req io.Copy 或 ioutil.ReadAll，或者尝试对大量对象的内存分配和释放的场景进行优化，这篇文章可能对你有帮助 2. 起因 在我们的一个程序中，使用库imroc/req请求后端的HTTP服务， 从HTTP响应读取结果 有如下调用关系 Resq.Bytes() -> Resq.ToBytes() -> ioutil.ReadAll(r io.Reader) -> Buff.ReadFrom(r io.Reader) 使用pprof收集内存累计分配情况可以发现大量的内存分配由Buffer.grow(n int) 触发 3. 原因分析 ReadFrom的源码如下 // MinRead is the minimum slice size passed to a Read call by // Buffer.ReadFrom. As long as the Buffer has at least MinRead bytes beyond // what is required to hold the contents of r, ReadFrom will not grow the // underlying buffer. const MinRead = 512 // ReadFrom reads data from r until EOF and appends it to the buffer, growing // the buffer as needed. The return value n is the number of bytes read. Any // error except io.EOF encountered during the read is also returned. If the // buffer becomes too large, ReadFrom will panic with ErrTooLarge. func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error) { b.lastRead = opInvalid for { i := b.grow(MinRead) m, e := r.Read(b.buf[i:cap(b.buf)]) if m < 0 { panic(errNegativeRead) } b.buf = b.buf[:i+m] n += int64(m) if e == io.EOF { return n, nil // e is EOF, so return nil explicitly } if e != nil { return n, e } } } 由于ioutil.ReadAll不知道最终需要多大的空间来存储结果数据，它采取的做法是，初始分配一个较小的Buff（大小为MinRead), 一边从输入中读取数据放入Buff, 一边看是否能够存下所有数据，如果不能则尝试扩大这个Buff(调用Buffer.grow()) ...