-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathrepo_cache_ft_nginx.html
313 lines (282 loc) · 29.8 KB
/
repo_cache_ft_nginx.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="canonical" href="http://tdt.rocks/repo_cache_ft_nginx.html" />
<title> TDT — Setting up a repo cache with NGINX
</title>
<link type="application/atom+xml" rel="alternate" href="/feed/atom.xml" title="TDT Atom Feed">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Montserrat" type="text/css">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css">
<link rel="stylesheet" href="http://tdt.rocks/theme/css/pygments.css">
<link rel="stylesheet" href="http://tdt.rocks/theme/css/style.css">
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-71190709-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-71190709-1');
</script>
<meta name="author" content="James Nzomo">
<meta name="description" content="A simple how to on setting up a linux package manager repo cache for those situations where a full blown mirror would be simply overkill.">
<meta name="tags" contents="nginx, proxy, cache, linux packages, ">
</head>
<body>
<header class="header">
<div class="container">
<div class="header-inner">
<!-- <h3 class="header-text">Thoughts & Deeds on Tech</h3> -->
<ul class="header-menu list-inline text-right">
<li><a class="nodec" href="http://tdt.rocks/categories.html">Categories</a></li>
<li class="muted">|</li>
<li><a class="nodec social-link icon-github" href="https://github.com/mrmoje" target="_blank"></a></li>
<li><a class="nodec social-link icon-twitter" href="https://twitter.com/mrmoje" target="_blank"></a></li>
<li><a class="nodec social-link icon-rss" href="http://tdt.rocks/feed/atom.xml" target="_blank"></a></li>
<li><a class="nodec social-link icon-stackoverflow" href="https://stackoverflow.com/users/1002644/moje" target="_blank"></a></li>
</ul>
<h1 class="header-name">
<a id="sitename" class="nodec" href="http://tdt.rocks">TDT</a>
</h1>
</div>
</div>
</header> <!-- /.header --> <div class="container">
<div class="post full-post">
<h1 class="post-title">
<a href="/repo_cache_ft_nginx.html" title="Permalink to Setting up a repo cache with NGINX">Setting up a repo cache with NGINX</a>
</h1>
<hr>
<ul class="list-inline">
<li class="post-date">
<a class="text-muted" href="/repo_cache_ft_nginx.html" title="2015-12-14T03:25:00+03:00">December 14, 2015</a>
</li>
<li class="muted">·</li>
<li class="post-category">
<a href="http://tdt.rocks/category/infrastructure.html">Infrastructure</a>
</li>
<li class="muted">·</li>
<li>
<a href="/tag/nginx.html">nginx</a>, <a href="/tag/proxy.html">proxy</a>, <a href="/tag/cache.html">cache</a>, <a href="/tag/linux packages.html">linux packages</a> </li>
<li class="muted">·</li>
<li>
<address class="post-author">
By <a href="http://tdt.rocks/author/james-nzomo.html">James Nzomo</a>
</address>
</li>
</ul>
<div class="post-content">
<h1>Intro</h1>
<p>Say you have a ton of Linux hosts package managed by apt or rpm. Maybe it's
a cluster, cloud env or just a simple learner's lab where a large groups of
(if not all) nodes will end up with the same or similar set of packages
installed and/or upgraded.</p>
<p>Local Linux repo mirrors would be a fantastic time & bandwidth saver for such
a setup...but the thought of dedicating double, triple or quadruple digit Gigs
(my <a href="pages/glossary#wag">WAG</a> as at Dec 2015 - depending on desired distros & architectures)
and the high probability that swathes of space will go wasted on packages
no one will ever install should be enough to make you seek a more optimal
option.</p>
<p>What you probably want is a good repository caching proxy that initially fetches
a pkg when a host has first dibs request, then stores and serves the same pkg
for subsequent matching requests.</p>
<h1>The Setup</h1>
<p>There's infinite ways to go about this. This post will cover how it's done using
webserver reverse proxy caching feature to handle requests on behalf of a few
upstream/origin repo servers.</p>
<p>For more info on the general idea/concept, see:-
<a href="(https://trafficserver.readthedocs.org/en/5.3.x/admin/reverse-proxy-http-redirects.en.html)">trafficserver</a></p>
<p>Though it's possible with
<a href="https://httpd.apache.org/docs/2.4/caching.html">Apache2</a>,
<a href="http://redmine.lighttpd.net/projects/1/wiki/Docs_ModCache">lighthttpd</a> or even
<a href="http://trafficserver.apache.org">trafficserver</a> we'll just go with
<a href="https://www.nginx.com/blog/nginx-caching-guide/">NGINX</a> for no apparent reason.</p>
<p>So start by grabbing yourself some webserver:- <code>apt-get install nginx</code> and we'll sort this
out in one simple sites-available config.</p>
<h3>The NGINX config</h3>
<p>The following minimal config sets nginx to handle this biz for the ubuntu
repos defined there in. Read on...this section continues inside the conf file as comments.</p>
<div class="highlight"><pre><span></span><code><span class="s s-Atom">#</span> <span class="nv">The</span> <span class="s s-Atom">upstream</span> <span class="s s-Atom">directive</span> <span class="s s-Atom">defining</span> <span class="s s-Atom">the</span> <span class="s s-Atom">backend</span><span class="o">/</span><span class="s s-Atom">upstream</span> <span class="s s-Atom">repo</span> <span class="s s-Atom">hosts</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">see</span> <span class="nn">http</span><span class="p">:</span><span class="o">//</span><span class="s s-Atom">nginx</span><span class="p">.</span><span class="s s-Atom">org</span><span class="o">/</span><span class="s s-Atom">en</span><span class="o">/</span><span class="s s-Atom">docs</span><span class="o">/</span><span class="s s-Atom">http</span><span class="o">/</span><span class="s s-Atom">ngx_http_upstream_module</span><span class="p">.</span><span class="s s-Atom">html</span>
<span class="s s-Atom">#</span> <span class="nv">There</span> <span class="o">is</span> <span class="s s-Atom">a</span> <span class="s s-Atom">primary</span> <span class="s s-Atom">and</span> <span class="s s-Atom">a</span> <span class="s s-Atom">backup</span><span class="p">.</span> <span class="nv">You</span> <span class="s s-Atom">can</span> <span class="s s-Atom">have</span> <span class="s s-Atom">more</span> <span class="s s-Atom">than</span> <span class="s s-Atom">that</span><span class="p">.</span>
<span class="s s-Atom">#</span> <span class="nv">NGINX</span> <span class="s s-Atom">will</span> <span class="s s-Atom">try</span> <span class="s s-Atom">to</span> <span class="s s-Atom">load</span> <span class="s s-Atom">balance</span> <span class="s s-Atom">between</span> <span class="s s-Atom">them</span> <span class="s s-Atom">using</span> <span class="s s-Atom">a</span> <span class="s2">"weighted round-robin"</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">algorithm</span><span class="p">.</span>
<span class="s s-Atom">#</span> <span class="nv">Most</span> <span class="s s-Atom">requests</span> <span class="s s-Atom">go</span> <span class="s s-Atom">to</span> <span class="s s-Atom">the</span> <span class="s s-Atom">primary</span><span class="p">,</span> <span class="s s-Atom">fewer</span> <span class="s s-Atom">to</span> <span class="s s-Atom">the</span> <span class="s s-Atom">rest</span><span class="p">.</span>
<span class="s s-Atom">#</span> <span class="nv">Error</span> <span class="s s-Atom">retries</span> <span class="s s-Atom">trickle</span> <span class="s s-Atom">down</span> <span class="s s-Atom">the</span> <span class="s s-Atom">server</span> <span class="s s-Atom">definitions</span><span class="p">.</span> <span class="nv">If</span> <span class="s s-Atom">the</span> <span class="s s-Atom">last</span> <span class="s s-Atom">errs</span><span class="p">,</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">the</span> <span class="s s-Atom">result</span> <span class="o">is</span> <span class="s s-Atom">passed</span> <span class="s s-Atom">on</span> <span class="s s-Atom">to</span> <span class="s s-Atom">the</span> <span class="s s-Atom">client</span><span class="p">.</span>
<span class="s s-Atom">#</span> <span class="nv">By</span> <span class="s s-Atom">default</span> <span class="nv">I</span> <span class="s s-Atom">found</span> <span class="s s-Atom">that</span> <span class="s s-Atom">if</span> <span class="s s-Atom">one</span> <span class="o">is</span> <span class="s s-Atom">unreachable</span> <span class="s s-Atom">during</span> <span class="s s-Atom">startup</span><span class="p">,</span> <span class="s s-Atom">nginx</span> <span class="s s-Atom">will</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">kaput</span> <span class="s s-Atom">leaving</span> <span class="s s-Atom">clues</span> <span class="s s-Atom">in</span> <span class="s s-Atom">the</span> <span class="s s-Atom">error</span> <span class="s s-Atom">log</span><span class="p">.</span>
<span class="s s-Atom">upstream</span> <span class="s s-Atom">ubuntu</span> <span class="p">{</span>
<span class="s s-Atom">server</span> <span class="s s-Atom">ke</span><span class="p">.</span><span class="s s-Atom">archive</span><span class="p">.</span><span class="s s-Atom">ubuntu</span><span class="p">.</span><span class="s s-Atom">com</span><span class="p">;</span>
<span class="s s-Atom">server</span> <span class="s s-Atom">archive</span><span class="p">.</span><span class="s s-Atom">ubuntu</span><span class="p">.</span><span class="s s-Atom">com</span> <span class="s s-Atom">backup</span><span class="p">;</span>
<span class="p">}</span>
<span class="s s-Atom">#</span> <span class="nv">The</span> <span class="s s-Atom">following</span> <span class="s s-Atom">directive</span> <span class="s s-Atom">configures</span> <span class="s s-Atom">the</span> <span class="s s-Atom">cache</span><span class="p">.</span> <span class="nv">Storage</span> <span class="s s-Atom">et</span> <span class="s s-Atom">all</span><span class="p">.</span> <span class="nv">You</span> <span class="s s-Atom">may</span> <span class="s s-Atom">want</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">the</span> <span class="s s-Atom">storage</span> <span class="s s-Atom">location</span> <span class="s s-Atom">at</span> <span class="s s-Atom">a</span> <span class="s s-Atom">mount</span> <span class="s s-Atom">point</span> <span class="s s-Atom">with</span> <span class="s s-Atom">a</span> <span class="s s-Atom">filesystem</span> <span class="s s-Atom">on</span> <span class="s s-Atom">separate</span> <span class="s s-Atom">storage</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">device</span><span class="o">/</span><span class="s s-Atom">backend</span> <span class="s s-Atom">to</span> <span class="s s-Atom">match</span><span class="p">.</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">see</span> <span class="nn">http</span><span class="p">:</span><span class="o">//</span><span class="s s-Atom">nginx</span><span class="p">.</span><span class="s s-Atom">org</span><span class="o">/</span><span class="s s-Atom">en</span><span class="o">/</span><span class="s s-Atom">docs</span><span class="o">/</span><span class="s s-Atom">http</span><span class="o">/</span><span class="s s-Atom">ngx_http_proxy_module</span><span class="p">.</span><span class="s s-Atom">html#proxy_cache_path</span><span class="p">.</span>
<span class="s s-Atom">proxy_cache_path</span> <span class="o">/</span><span class="s s-Atom">var</span><span class="o">/</span><span class="s s-Atom">repo_mirror</span> <span class="s s-Atom">#</span> <span class="s s-Atom">defines</span> <span class="s s-Atom">where</span> <span class="s s-Atom">the</span> <span class="s s-Atom">cache</span> <span class="o">is</span> <span class="s s-Atom">stashed</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">defines</span> <span class="s s-Atom">cache</span> <span class="s s-Atom">path</span> <span class="nf">heirarchy</span> <span class="p">(</span><span class="s s-Atom">yaani</span> <span class="s s-Atom">num</span> <span class="s s-Atom">directory</span> <span class="s s-Atom">levels</span> <span class="s s-Atom">in</span> <span class="s s-Atom">cache</span><span class="p">)</span>
<span class="s s-Atom">levels</span><span class="o">=</span><span class="mi">1</span><span class="s s-Atom">:</span><span class="mi">2</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">defines</span> <span class="s s-Atom">name</span> <span class="s s-Atom">and</span> <span class="s s-Atom">size</span> <span class="s s-Atom">of</span> <span class="s s-Atom">zone</span> <span class="s s-Atom">where</span> <span class="s s-Atom">all</span> <span class="s s-Atom">cache</span> <span class="s s-Atom">keys</span> <span class="s s-Atom">and</span> <span class="s s-Atom">cache</span> <span class="s s-Atom">metadata</span> <span class="s s-Atom">are</span> <span class="s s-Atom">stashed</span><span class="p">.</span>
<span class="s s-Atom">#</span> <span class="nv">Servers</span> <span class="s s-Atom">as</span> <span class="s s-Atom">a</span> <span class="s s-Atom">lookup</span> <span class="s s-Atom">for</span> <span class="s s-Atom">cached</span> <span class="s s-Atom">data</span>
<span class="s s-Atom">keys_zone</span><span class="o">=</span><span class="s s-Atom">repository_cache:</span><span class="mi">50</span><span class="s s-Atom">m</span>
<span class="s s-Atom">#</span> <span class="nv">The</span> <span class="s s-Atom">cached</span> <span class="s s-Atom">data</span> <span class="s s-Atom">access</span> <span class="s s-Atom">timeout</span><span class="p">.</span> <span class="nv">Pkgs</span> <span class="s s-Atom">get</span> <span class="s s-Atom">nuked</span> <span class="s s-Atom">if</span> <span class="s s-Atom">no</span> <span class="s s-Atom">access</span> <span class="s s-Atom">in</span> <span class="mi">14</span> <span class="s s-Atom">days</span><span class="p">.</span>
<span class="s s-Atom">inactive</span><span class="o">=</span><span class="mi">14</span><span class="s s-Atom">d</span>
<span class="s s-Atom">#</span> <span class="nv">Cache</span> <span class="s s-Atom">size</span> <span class="s s-Atom">limit</span>
<span class="s s-Atom">max_size</span><span class="o">=</span><span class="mi">100</span><span class="s s-Atom">g</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="nv">Our</span> <span class="s s-Atom">no</span><span class="o">-</span><span class="s s-Atom">name</span> <span class="s s-Atom">server</span> <span class="s s-Atom">block</span>
<span class="s s-Atom">server</span> <span class="p">{</span>
<span class="s s-Atom">#</span> <span class="nv">Keep</span> <span class="s s-Atom">our</span> <span class="s s-Atom">eyes</span> <span class="s s-Atom">peeled</span> <span class="s s-Atom">on</span> <span class="s s-Atom">port</span> <span class="mi">80</span>
<span class="s s-Atom">listen</span> <span class="mi">80</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="nv">Location</span> <span class="s s-Atom">directive</span> <span class="s s-Atom">for</span> <span class="s s-Atom">the</span> <span class="o">/</span><span class="s s-Atom">ubuntu</span> <span class="s s-Atom">path</span>
<span class="s s-Atom">location</span> <span class="o">/</span><span class="s s-Atom">ubuntu</span> <span class="p">{</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">our</span> <span class="s s-Atom">cache</span><span class="err">'</span><span class="s s-Atom">s</span> <span class="s s-Atom">root</span>
<span class="s s-Atom">root</span> <span class="o">/</span><span class="s s-Atom">var</span><span class="o">/</span><span class="s s-Atom">repo_mirror</span><span class="o">/</span><span class="s s-Atom">index_data</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">look</span> <span class="s s-Atom">for</span> <span class="s s-Atom">packages</span> <span class="s s-Atom">in</span> <span class="s s-Atom">the</span> <span class="s s-Atom">following</span> <span class="s s-Atom">order</span>
<span class="s s-Atom">try_files</span> <span class="err">$</span><span class="s s-Atom">uri</span> <span class="s s-Atom">@ubuntu</span><span class="p">;</span>
<span class="p">}</span>
<span class="s s-Atom">#</span> <span class="nv">Location</span> <span class="s s-Atom">directive</span> <span class="s s-Atom">for</span> <span class="s s-Atom">the</span> <span class="s s-Atom">named</span> <span class="s s-Atom">location</span> <span class="s s-Atom">defined</span> <span class="s s-Atom">above</span>
<span class="s s-Atom">location</span> <span class="s s-Atom">@ubuntu</span> <span class="p">{</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">map</span> <span class="s s-Atom">this</span> <span class="s s-Atom">to</span> <span class="s s-Atom">the</span> <span class="s s-Atom">upstream</span> <span class="s s-Atom">definition</span> <span class="s s-Atom">above</span>
<span class="s s-Atom">proxy_pass</span> <span class="nn">http</span><span class="p">:</span><span class="o">//</span><span class="s s-Atom">ubuntu</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="nv">Update</span> <span class="mi">20201127</span><span class="p">,</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">for</span> <span class="s s-Atom">some</span> <span class="s s-Atom">good</span> <span class="s s-Atom">measure</span><span class="p">,</span> <span class="s s-Atom">set</span> <span class="s s-Atom">the</span> <span class="s s-Atom">host</span> <span class="nf">header</span> <span class="p">(</span><span class="s s-Atom">just</span> <span class="s s-Atom">incase</span> <span class="s s-Atom">the</span> <span class="s s-Atom">upstream</span> <span class="s s-Atom">insists</span> <span class="s s-Atom">on</span> <span class="s s-Atom">it</span><span class="p">)</span>
<span class="s s-Atom">#</span> <span class="nv">See</span><span class="p">:-</span>
<span class="s s-Atom">#</span> <span class="o">-</span> <span class="nn">https</span><span class="p">:</span><span class="o">//</span><span class="s s-Atom">bugs</span><span class="p">.</span><span class="s s-Atom">launchpad</span><span class="p">.</span><span class="s s-Atom">net</span><span class="o">/</span><span class="s s-Atom">ubuntu/+source</span><span class="o">/</span><span class="s s-Atom">apt/+bug</span><span class="o">/</span><span class="mi">1838771</span>
<span class="s s-Atom">#</span> <span class="o">-</span> <span class="nn">https</span><span class="p">:</span><span class="o">//</span><span class="s s-Atom">twitter</span><span class="p">.</span><span class="s s-Atom">com</span><span class="o">/</span><span class="nv">Zulgrib</span><span class="o">/</span><span class="s s-Atom">status</span><span class="o">/</span><span class="mi">1324146475685826561</span>
<span class="s s-Atom">#</span> <span class="nv">How</span> <span class="s s-Atom">to</span> <span class="s s-Atom">cater</span> <span class="s s-Atom">for</span> <span class="s s-Atom">the</span> <span class="s s-Atom">backup</span> <span class="nv">I</span> <span class="s s-Atom">hear</span> <span class="s s-Atom">you</span> <span class="s s-Atom">ask?</span> <span class="p">...</span><span class="s s-Atom">well</span> <span class="s s-Atom">thats</span> <span class="s s-Atom">an</span> <span class="s s-Atom">exercise</span> <span class="s s-Atom">for</span> <span class="s s-Atom">the</span> <span class="s s-Atom">reader</span> <span class="p">;)</span>
<span class="s s-Atom">proxy_set_header</span> <span class="nv">Host</span> <span class="s s-Atom">ke</span><span class="p">.</span><span class="s s-Atom">archive</span><span class="p">.</span><span class="s s-Atom">ubuntu</span><span class="p">.</span><span class="s s-Atom">com</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="mi">14</span><span class="s s-Atom">days</span> <span class="s s-Atom">of</span> <span class="s s-Atom">caching</span> <span class="s s-Atom">for</span> <span class="s s-Atom">http</span> <span class="s s-Atom">code</span> <span class="mi">200</span> <span class="s s-Atom">response</span> <span class="s s-Atom">content</span>
<span class="s s-Atom">proxy_cache_valid</span> <span class="mi">200</span> <span class="mi">14</span><span class="s s-Atom">d</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">we</span> <span class="s s-Atom">set</span> <span class="s s-Atom">our</span> <span class="s2">"repository_cache"</span> <span class="s s-Atom">zone</span> <span class="s s-Atom">for</span> <span class="s s-Atom">caching</span>
<span class="s s-Atom">proxy_cache</span> <span class="s s-Atom">repository_cache</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="nv">Use</span> <span class="s s-Atom">stale</span> <span class="s s-Atom">cached</span> <span class="s s-Atom">data</span> <span class="s s-Atom">in</span> <span class="s s-Atom">the</span> <span class="s s-Atom">error</span> <span class="s s-Atom">events</span> <span class="s s-Atom">defined</span>
<span class="s s-Atom">proxy_cache_use_stale</span> <span class="s s-Atom">error</span> <span class="s s-Atom">timeout</span> <span class="s s-Atom">invalid_header</span> <span class="s s-Atom">updating</span><span class="p">;</span>
<span class="s s-Atom">#</span> <span class="s s-Atom">pass</span> <span class="s s-Atom">request</span> <span class="s s-Atom">to</span> <span class="nf">next</span> <span class="p">(</span><span class="s s-Atom">backup</span><span class="p">)</span> <span class="s s-Atom">server</span> <span class="s s-Atom">in</span> <span class="s s-Atom">the</span> <span class="s s-Atom">error</span> <span class="s s-Atom">events</span> <span class="s s-Atom">defined</span>
<span class="s s-Atom">proxy_next_upstream</span> <span class="s s-Atom">error</span> <span class="s s-Atom">timeout</span> <span class="s s-Atom">invalid_header</span> <span class="s s-Atom">http_500</span> <span class="s s-Atom">http_502</span> <span class="s s-Atom">http_503</span> <span class="s s-Atom">http_504</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>The above config is by no means comprehensive but it works good enough as a demo. Feel free
to copy paste it into <code>/etc/nginx/sites-available/</code>.</p>
<p>One enhancement that can be made to all this is to configure a reliable storage device for
cache store.</p>
<h3>The client side</h3>
<p>It goes without saying that repo related requests have to somehow be routed to the
proxy in order for all this to work.</p>
<p>One way is to modify the client's <code>/etc/apt/sources.list</code> or <code>/etc/yum.repos.d/CentOS-*.repo</code>
to point to our server.</p>
<p>Another would be to try intercept requests using various forms of MITM sorcery
like forging faux DNS replies for repo FQDNs or router configs/rules.</p>
<p>Since the client hosts can be configured in most cases and the latter
option is in grey area territory, we'll go with the former and modify the
repo configs. Read on to the next section for that (because the writer wrote
that before the one you reading now and he cannot be bothered to repeat
himself or make mods to the article at this point).</p>
<h1>Bonus vagrant-ansible deploy material</h1>
<p>Just in case you'd like to test all this, or would like some CM to deploy the same
IRL, there is a rudimentary
<a href="https://github.com/mrmoje/repo_cache_plays_staring_nginx">"demo setup git repo"</a>
complete with ansible plays and a four host vagrant playground - one "repo-cacher"
and three clients - ubuntu trusty, debian jessie and CentOS 7 - which upon successful
<code>vagrant up</code>, will be provisioned to fetch their stuff from the repo-cacher.</p>
<p>To try it out! In case you haven't already, feel free to install latest:-
<a href="https://www.vagrantup.com/downloads">Vagrant</a>,
<a href="https://docs.ansible.com/ansible/intro_installation.html">Ansible</a> and
<a href="https://www.virtualbox.org/wiki/Downloads">virtualbox</a>.
The whole #! will take a while to install, especially if running <code>vagrant up XXXXX</code>
for the first time and from a poor connection, but IMHO similar IRL bare metal
setup would probably take much longer to depoy.</p>
<p>Details on how the vagrant and ansible stuff comes together will be blawged in a future
post. For now, checkout the (currently empty)
<a href="https://github.com/mrmoje/repo_cache_plays_staring_nginx/blob/master/README.md">README</a>
or just run <code>vagrant up repo-cache</code> to launch the repo cache then <code>vagrant ssh repo-cache</code>
and open <code>/etc/nginx/sites-available/repo-cache.conf</code>.</p>
<p>Have a look at the tri-distro proxy cache config and feel free to mod as you please.
Reset it all by running <code>vagrant provision repo-cache</code></p>
<p>You should also be able to browse the upstream repos via the proxy from the following links:-</p>
<ul>
<li><a href="http://repo-cache.local/ubuntu/">http://repo-cache.local/ubuntu/</a></li>
<li><a href="http://repo-cache.local/debian/">http://repo-cache.local/debian/</a></li>
<li><a href="http://repo-cache.local/centos/">http://repo-cache.local/centos/</a></li>
</ul>
<p><strong>NB</strong>: Replace the hostname with <code>10.64.0.2</code> if you dont have a mdns client installed.</p>
<p>You can subsequently launch one, two or all of the other client hosts
<code>vagrant up repo-cache ubuntu #(or debian or centos)</code>. Feel free to <code>vagrant ssh</code> into
them, take a look at <code>/etc/apt/sources.list</code> or <code>/etc/yum.repos.d/CentOS-*.repo</code> and
give some package installs a run. Don't forget to peek at <code>repo-cache:/var/repo_mirror</code>
while you do that.</p>
<p>Look at the <a href="https://github.com/mrmoje/repo_cache_plays_ft_nginx/blob/master/Vagrantfile">Vagrant</a>
file to see hodge-podge of provisioning methods and the
<a href="https://www.gnu.org/software/sed/manual/sed.html">sedding</a> that swaps the vanilla repos
for our example proxy.</p>
<h1>Outro</h1>
<p>We put a <a href="https://github.com/kili/playbooks/blob/master/playbooks/repository_cache.yaml">variant of this proxy-cache</a>
method to good use at a cloud compute startup I used to work for and it worked
out well for us and our tenants.</p>
<p>In the end amongst other benefits, we were (obviously) able to reduce latency and
network traffic (by how much is anyones guess. We didn't care measure).
We also ended up with more space to put to good use. Space that would have otherwise
been taken up by a multi repo mirror.</p>
<p>The only possible downside I can fathom for this is that it will still
take longer for the first host/request to get sorted which may be bad for
the installing party depending on how pressed for time they are, but then again
with life, every now and then, somebody has to take one for the team. C'est la Vie!</p>
<h3>Resources, Refs & google juice:-</h3>
<ul>
<li><a href="(https://trafficserver.readthedocs.org/en/5.3.x/admin/reverse-proxy-http-redirects.en.html)">Trafficserver dox on reverse proxy</a></li>
<li><a href="(http://nginx.org/en/docs/http/ngx_http_proxy_module.html)">NGINX proxy module documentation</a></li>
<li>An excelent tut on caching with nginx with video & docker hands-on:- <a href="http://czerasz.com/2015/03/30/nginx-caching-tutorial/">http://czerasz.com/2015/03/30/nginx-caching-tutorial/</a></li>
</ul>
</div>
</div>
<hr class="separator">
<div>
<div id="disqus_thread">
<script>
var disqus_shortname = 'tdtrocks';
(function() {
var dsq = document.createElement('script');
dsq.type = 'text/javascript';
dsq.async = true;
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] ||
document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<noscript>
Please enable JavaScript to view the
<a href="https://disqus.com/?ref_noscript=tdtrocks">
comments powered by Disqus.
</a>
</noscript>
<a href="https://disqus.com" class="dsq-brlink">
blog comments powered by <span class="logo-disqus">Disqus</span>
</a>
</div>
</div>
</div>
<footer class="footer">
<div class="container">
<p class="text-center">
©Twenny Nineteen - James Nzomo, <a href="" target="_blank"></a> unless otherwise noted.
</p>
<div class="text-center">
Generated by <a href="http://getpelican.com" target="_blank">Pelican</a>
with a <a href="http://github.com/mrmoje/pelican-alchemy" target="_blank">copy-pasta</a>
of <a href="https://nairobilug.or.ke" target="_blank" >NairobiLUG's</a>
<a href="http://github.com/nairobilug/pelican-alchemy" target="_blank">alchemy</a> theme, <br/>
which inturn is a copy-pasta of Porter's <a href="https://github.com/porterjamesj/crowsfoot" target="_blank">crowsfoot</a> theme. <br/>
--- end of <a href="//tdt.rocks/pages/glossary#google-juice">google juice</a> ---.
</div>
</div>
</footer> <!-- /.footer -->
<script src="http://tdt.rocks/theme/js/jquery.min.js"></script>
<script src="http://tdt.rocks/theme/js/bootstrap.min.js"></script>
</body> <!-- 42 -->
</html>