|
1 | 1 | # NEKit |
2 | 2 |
|
3 | | -[](https://gitter.im/zhuhaow/NEKit?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [](https://telegram.me/NEKitGroup) [](https://travis-ci.org/zhuhaow/NEKit) [](https://github.com/zhuhaow/NEKit/releases) [](https://codecov.io/gh/zhuhaow/NEKit) [](https://github.com/Carthage/Carthage) [](LICENSE.md) |
| 3 | +NEKit is deprecated. |
4 | 4 |
|
5 | | -**Take a look at the next generation of this project, [libnekit](https://github.com/zhuhaow/libnekit).** |
| 5 | +It should still work but I'm not intent on maintaining it anymore. It has many |
| 6 | +flaws and needs a revamp to be a high-quality library. |
6 | 7 |
|
7 | | -A toolkit for Network Extension Framework. |
| 8 | +The architecture of NEKit is not ideal due to my lack of experience when I was |
| 9 | +writing it. To make it worse, it's not in its most efficient form as there |
| 10 | +wasn't a good async io library while people abuse it for simple features that |
| 11 | +should be just implemented directly or use other lightweight libraries. |
8 | 12 |
|
9 | | -NEKit is the successor of [Soca](https://github.com/zhuhaow/soca-ios). The main goal of NEKit is to provide things needed in building a Network Extension app with `NETunnelProvider` to bypass network filtering and censorship while keep the framework as non-opinionated as possible. |
| 13 | +NEKit is not, was never intended, and probably will never become a library that |
| 14 | +simply works correctly out-of-the-box without understanding what it is doing |
| 15 | +under the hook. Through all these years, one thing I learned is for some function |
| 16 | +this low level, the developer should understand what oneself is doing. I'm |
| 17 | +always concerned that people are creating apps that slowing down users' phones |
| 18 | +unnecessarily because of this library and I feel responsible. |
10 | 19 |
|
11 | | -**NEKit does not depend on Network Extension framework. You can use NEKit without Network Extension entitlement to build a rule based proxy in a few lines.** |
12 | | - |
13 | | -**Do not enable `TUNInterface` as of now since it is not working.** |
14 | | - |
15 | | -There are two demos that you should check out. |
16 | | - |
17 | | -[SpechtLite](https://github.com/zhuhaow/SpechtLite) does not require Network Extension and anyone can use it. |
18 | | - |
19 | | -[Specht](https://github.com/zhuhaow/Specht) is another demo that requires Network Extension entitlement. |
20 | | - |
21 | | -Currently, NEKit supports: |
22 | | - |
23 | | -- Forward requests through different proxies based on remote host location, remote host domain or the connection speed of proxies. |
24 | | -- Integrated tun2socks framework to reassemble TCP packets into TCP flows. |
25 | | -- A DNS server that rewrites request and response. |
26 | | -- Some tools to build IP packets. |
27 | | -- ... |
28 | | - |
29 | | -**Part of the following content may be out of date. But the general design philosophy still holds.** |
30 | | -Check document [here](https://zhuhaow.github.io/NEKit), which should be up to date. |
31 | | - |
32 | | -Also, you may be more interested in [Potatso](https://github.com/shadowsocks/Potatso-iOS) if you just need an open source iOS app with GUI supporting shadowsocks. |
33 | | - |
34 | | -**[Wingy](https://itunes.apple.com/cn/app/wingy-mian-fei-banvpn-ke-hu/id1148026741?mt=8) is a free app built with NEKit available on App Store for your iDevice.** Note Wingy is not created by or affiliated with me. |
35 | | - |
36 | | -If you have any questions (not bug report), **please join Gitter or Telegram instead of opening an issue**. |
37 | | - |
38 | | -## Principle |
39 | | - |
40 | | -NEKit tries to be as flexible and non-opionated as possible. |
41 | | - |
42 | | -However, it is not always as modular as you may think if you want to reproduce transport layer from network layer. |
43 | | - |
44 | | -NEKit follows one fundamental principle to keep the best network performance: The device connecting to target server directly resolves the domain. |
45 | | - |
46 | | -This should not be a problem if the applications on your device connect to the local proxy server directly, where we can get the request domain information then send that to remote proxy server if needed. |
47 | | - |
48 | | -But consider that if an application tries to make a socket connection by itself, which generally consists of two steps: |
49 | | - |
50 | | -1. Make a DNS lookup to find the IP address of the target server. |
51 | | -2. Connect to the remote server by socket API provided by the system. |
52 | | - |
53 | | -We can read only two independent things from the TUN interface, a UDP packet containing the DNS lookup request and a TCP flow consisting of a serial of TCP packets. So there is no way we can know the initial request domain for the TCP flow. And since there may be multiple domains served on the same host, we can not get the origin domain by saving the DNS response and looking that up reversely later. |
54 | | - |
55 | | -The only solution is to create a fake IP pool and assign each requested domain with a unique fake IP so we can look that up reversely. Every connection needs to look that up from the DNS server afterwards; this is the only non-modular part of NEKit which is already encapsulated in `ConnectSession`. |
56 | | - |
57 | | -## Usage |
58 | | - |
59 | | -### Add it to you project |
60 | | -I recommend adding this project to your project, which is easier to debug. |
61 | | - |
62 | | -However, you can still use it with Carthage (you'll need Carthage anyway since NEKit uses Carthage) by adding |
63 | | -```ruby |
64 | | -github "zhuhaow/NEKit" |
65 | | -``` |
66 | | -to you `Cartfile`. |
67 | | - |
68 | | -Use |
69 | | -```carthage update --no-use-binaries --platform mac,ios``` |
70 | | -to install all frameworks. Do not use pre-compiled binaries since some of them might be buggy. |
71 | | - |
72 | | -### Overview |
73 | | -NEKit basically consists of two parts, a proxy server forwarding socket data based on user defined rules and an IP stack reassembling IP packets back to TCP flow as a socket. |
74 | | - |
75 | | -### Rule manager |
76 | | -Before starting any proxy server, we need to define rules. |
77 | | - |
78 | | -Each rule consists of two parts, one defining what kinding of request matches this rule and the other defining what adapter to use. An adapter represents the abstraction of a socket connection to a remote proxy server (or remote host). We use `AdapterFactory` to build adapters. |
79 | | - |
80 | | -NEKit provides `AdapterSocket` supporting HTTP/HTTPS/SOCK5 proxy and Shadowsocks(AES-128-CFB/AES-192-CFB/AES-256-CFB/chacha20/salsa20/rc4-md5). Let me know if there is any other type of proxy needed. You can also implement your own `AdapterSocket`. |
81 | | - |
82 | | -```swift |
83 | | -// Define remote adapter first |
84 | | -let directAdapterFactory = DirectAdapterFactory() |
85 | | -let httpAdapterFactory = HTTPAdapterFactory(serverHost: "remote.http.proxy", serverPort: 3128, auth: nil) |
86 | | -let ssAdapterFactory = ShadowsocksAdapterFactory(serverHost: "remote.ss.proxy", serverPort: 7077, encryptMethod: "AES-256-CFB", password: "1234567890") |
87 | | - |
88 | | -// Then create rules |
89 | | -// To use geo ip database, first init it |
90 | | -// Download the binary database https://dev.maxmind.com/geoip/geoip2/geolite2/ |
91 | | -GeoIP.database = MMDB("PATH_TO_THE_DATABASE")! |
92 | | -let chinaRule = CountryRule(countryCode: "CN", match: true, adapterFactory: directAdapterFactory) |
93 | | -// `urls` are regular expressions |
94 | | -let listRule = try! ListRule(adapterFactory: ssAdapterFactory, urls: ["some\\.site\\.does\\.not\\.exists"]) |
95 | | -let allRule = AllRule(adapterFactory: httpAdapterFactory) |
96 | | - |
97 | | -// Create rule manager, rules will be matched in order. |
98 | | -let manager = RuleManager(fromRules: [listRule, chinaRule, allRule], appendDirect: true) |
99 | | - |
100 | | -// Set this manager as the active manager |
101 | | -RuleManager.currentManager = ruleManager |
102 | | -``` |
103 | | - |
104 | | -There is also `Configuration` to load rules from a Yaml config file. But that is not recommended. |
105 | | - |
106 | | - |
107 | | -### Proxy server |
108 | | -Now we can start a HTTP/SOCKS5 proxy server locally. |
109 | | - |
110 | | -```swift |
111 | | -let server = GCDHTTPProxyServer(address: IPv4Address(fromString: "127.0.0.1"), port: Port(port: 9090)) |
112 | | -try! server.start() |
113 | | -``` |
114 | | - |
115 | | -Now there is a HTTP proxy server running on `127.0.0.1:9090` which will forward requests based on rules defined in `RuleManager.currentManager`. |
116 | | - |
117 | | -If you do not want to handle IP packets, then that's it, just set the proxy to `127.0.0.1:9090` in System Preferences and you are good to go. |
118 | | - |
119 | | -If you want to read on, you will have to request Network Extention entitlement from Apple. |
120 | | - |
121 | | -But even if you use NetworkExtention to set up the network proxy, it does not mean you have to process packets, just do not route anything to the TUN interface and do not set up `IPStack`. For iOS, if you claim you have implemented it but just do nothing the users probably will never notice. |
122 | | - |
123 | | -### IP stack |
124 | | - |
125 | | -Assuming you already set up a working extension with correct routing configurations (Google how, this is not trivial). |
126 | | - |
127 | | -In |
128 | | - |
129 | | -```swift |
130 | | -startTunnelWithOptions(options: [String : NSObject]?, completionHandler: (NSError?) -> Void) |
131 | | -``` |
132 | | -set `RuleManager` and start proxy server(s) and then create an instance representing the TUN interface by |
133 | | - |
134 | | -```swift |
135 | | -let stack = TUNInterface(packetFlow: packetFlow) |
136 | | -``` |
137 | | - |
138 | | -We also have to set |
139 | | - |
140 | | -```swift |
141 | | -RawSocketFactory.TunnelProvider = self |
142 | | -``` |
143 | | -to create socket to connect to remote servers with `NETunnelProvider`. |
144 | | - |
145 | | -Then we need to register ip stacks implementing `IPStackProtocol` to process IP packets. |
146 | | - |
147 | | -NEKit provides several stacks. |
148 | | - |
149 | | -#### TCPStack |
150 | | -`TCPStack` process TCP packets and reassembles them back into TCP flows then send them to the proxy server specified by `proxyServer` variable. You have to set `proxyServer` before registering `TCPStack` to `TUNInterface`. |
151 | | - |
152 | | -#### DNSServer |
153 | | -DNS server is implemented as an IP stack processing UDP packets send to it. |
154 | | - |
155 | | -First create an DNS server with a fake IP pool. (You should use fake IP, but you can disable it if you want to by set it to nil.) |
156 | | - |
157 | | -```swift |
158 | | -let fakeIPPool = IPv4Pool(start: IPv4Address(fromString: "172.169.1.0"), end: IPv4Address(fromString: "172.169.255.0")) |
159 | | -let dnsServer = DNSServer(address: IPv4Address(fromString: "172.169.0.1"), port: Port(port: 53), fakeIPPool: fakeIPPool) |
160 | | -``` |
161 | | - |
162 | | -Then we have to define how to resolve the DNS requests, NEKit provides the most trivial one which sends the request to remote DNS server directly with UDP protocol, you can do anything you want by implementing `DNSResolverProtocol`. |
163 | | - |
164 | | -```swift |
165 | | -let resolver = UDPDNSResolver(address: IPv4Address(fromString: "114.114.114.114"), port: Port(port: 53)) |
166 | | -dnsServer.registerResolver(resolver) |
167 | | -``` |
168 | | - |
169 | | -It is very important to set |
170 | | - |
171 | | -```swift |
172 | | -DNSServer.currentServer = dnsServer |
173 | | -``` |
174 | | -so we can look up the fake IP reversely. |
175 | | - |
176 | | -#### UDPDirectStack |
177 | | -`UDPDirectStack` sends and reads UDP packets to and from remote server directly. |
178 | | - |
179 | | - |
180 | | -You can register these stack to TUN interface by |
181 | | - |
182 | | -```swift |
183 | | -interface.registerStack(dnsServer) |
184 | | -// Note this sends out every UDP packets directly so this must comes after any other stack that processes UDP packets. |
185 | | -interface.registerStack(UDPDirectStack()) |
186 | | -interface.registerStack(TCPStack.stack) |
187 | | -``` |
188 | | - |
189 | | -When everything is set up, you should start processing packets by calling `interface.start()` in the completion handler of `setTunnelNetworkSettings`. |
190 | | - |
191 | | -## Event |
192 | | - |
193 | | -You can use `Observer<T>` to observe the events in proxy servers and sockets. Check out observers in `DebugObserver.swift` as an example. |
194 | | - |
195 | | -## Dive in |
196 | | - |
197 | | -### Framework overview |
198 | | -The structure of the proxy server is given as follows: |
199 | | - |
200 | | -``` |
201 | | -┌──────────────────────────────────────────────────┐ |
202 | | -│ │ |
203 | | -│ ProxyServer │ |
204 | | -│ │ |
205 | | -├──────┬───┬──────┬───┬──────┬───┬──────┬───┬──────┤ |
206 | | -│Tunnel│ │Tunnel│ │Tunnel│ │Tunnel│ │Tunnel│ |
207 | | -└──────┘ └──────┘ └───▲──┘ └──────┘ └──────┘ |
208 | | - ╱ ╲ |
209 | | - ╱───────╱ ╲─────╲ |
210 | | - ╱ ╲ |
211 | | - ┌────────▼────────┐ ┌────────▼────────┐ |
212 | | - │ ProxySocket │ │ AdapterSocket │ |
213 | | - └────────▲────────┘ └────────▲────────┘ |
214 | | - │ │ |
215 | | - │ │ |
216 | | - ┌────────▼────────┐ ┌────────▼────────┐ |
217 | | - │RawSocketProtocol│ │RawSocketProtocol│ |
218 | | - └────────▲────────┘ └────────▲────────┘ |
219 | | - │ │ |
220 | | - │ │ |
221 | | - ╔══════▼═══════╗ ╔═══════▼══════╗ |
222 | | - ║ LOCAL ║ ║ REMOTE ║ |
223 | | - ╚══════════════╝ ╚══════════════╝ |
224 | | -``` |
225 | | - |
226 | | -When a new socket is accepted from the listening socket of the proxy server, it is wrapped in some implementation of `RawSocketProtocol` as a raw socket which just reads and writes data. |
227 | | - |
228 | | -Then it is wrapped in a subclass of `ProxySocket` which encapsulates the proxy logic. |
229 | | - |
230 | | -The `TCPStack` wraps the reassembled TCP flow (`TUNTCPSocket`) in `DirectProxySocket` then send it to the `mainProxy` server. |
231 | | - |
232 | | -Similarly, `AdapterSocket` encapsulated the logic of how to connect to remote and process the data flow. |
233 | | - |
234 | | -Pretty much everything of NEKit follows the [delegation pattern](https://en.wikipedia.org/wiki/Delegation_pattern). If you are not familiar with that, you should learn it first, probabaly by learning how to use [CocoaAsyncSocket](https://github.com/robbiehanson/CocoaAsyncSocket) (note that the sockets in NEKit are **not** thread-safe which is different from GCDAsyncSocket). |
235 | | - |
236 | | -### The lifetime of a `Tunnel` |
237 | | - |
238 | | -When a `RawSocketProtocol` socket is accepted or created by `TCPStack`, it is wrapped in a `ProxySocket` then in a `Tunnel`. The `Tunnel` will call `proxySocket.openSocket()` to let the proxy socket start process data. |
239 | | - |
240 | | -When the `ProxySocket` read enough data to build a `ConnectSession`, it calls `func didReceiveRequest(request: ConnectSession, from: ProxySocket)` of the `delegate` (which should be the `Tunnel`). |
241 | | - |
242 | | -The `Tunnel` then matches this request in `RuleManager` to get the corresponding `AdapterFactory`. Then `func openSocketWithSession(session: ConnectSession)` of the produced `AdapterSocket` is called to connect to remote server. |
243 | | - |
244 | | -The `AdapterSocket` calls `func didConnect(adapterSocket: AdapterSocket, withResponse response: ConnectResponse)` of the `Tunnel` to let the `ProxySocket` has a chance to respond to remote response. (This is ignored as of now.) |
245 | | - |
246 | | -Finally, when the `ProxySocket` and `AdapterSocket` are ready to forward data, they should call `func readyToForward(socket: SocketProtocol)` of the `Tunnel` to let it know. When both sides are ready, the tunnel will read from both sides and then send the received data to the other side intact. |
247 | | - |
248 | | -When any side of the tunnel is disconnected, the `func didDisconnect(socket: SocketProtocol)` is called then both sides are closed actively. The `Tunnel` will be released when both sides disconnect successfully. |
249 | | - |
250 | | - |
251 | | -## TODO |
252 | | -- [ ] Documents. |
253 | | -- [ ] IPv6 support. |
| 20 | +Thanks for everyone who has contributed, used or interested in this library. |
254 | 21 |
|
255 | 22 | ## License |
| 23 | + |
256 | 24 | Copyright (c) 2016, Zhuhao Wang |
257 | 25 | All rights reserved. |
258 | 26 |
|
259 | 27 | Redistribution and use in source and binary forms, with or without |
260 | 28 | modification, are permitted provided that the following conditions are met: |
261 | 29 |
|
262 | | -* Redistributions of source code must retain the above copyright notice, this |
| 30 | +- Redistributions of source code must retain the above copyright notice, this |
263 | 31 | list of conditions and the following disclaimer. |
264 | 32 |
|
265 | | -* Redistributions in binary form must reproduce the above copyright notice, |
| 33 | +- Redistributions in binary form must reproduce the above copyright notice, |
266 | 34 | this list of conditions and the following disclaimer in the documentation |
267 | 35 | and/or other materials provided with the distribution. |
268 | 36 |
|
269 | | -* Neither the name of NEKit nor the names of its |
| 37 | +- Neither the name of NEKit nor the names of its |
270 | 38 | contributors may be used to endorse or promote products derived from |
271 | 39 | this software without specific prior written permission. |
272 | 40 |
|
|
0 commit comments