Sinopia with Nginx and Proxy

[akshayakrsh]

Hi,

I have setup a Sinopia server with Nginx installed on it. My nginx configuration looks like this:

server {
  listen  80 default_server;
  location / {
    proxy_pass         http://127.0.0.1:4873;
  }
}

and my sinopia server is running via pm2 => pm2 start which sinopia -f. Lastly, my config yaml is a copy of https://github.com/rlidwka/sinopia/blob/master/conf/full.yaml (with web version enabled).

I am facing 2 problems:

1. Web Interface: The web interface of sinopia server when i hit the url http://server_ip/ is completely black and white. See the screenshot below to see what I mean.

2. NPM Registry lookup: The sinopia server is unable to do lookup in the official npm registry. I have already added registry on my machine using npm set registry http://SERVER_URL/sinopia but after that when I run npm install <module_name> e.g. "qs", it fails! The verbose error log is:

npm install qs --verbose
npm info it worked if it ends with ok
npm verb cli [ 'node', '/usr/local/bin/npm', 'install', 'qs', '--verbose' ]
npm info using npm@2.13.0
npm info using node@v0.12.7
npm verb node symlink /usr/local/bin/node
npm verb install initial load of /Users/akshayakumarsharma/Workspace/myapp/package.json
npm verb readDependencies loading dependencies from /Users/akshayakumarsharma/Workspace/myapp/package.json
npm verb cache add spec qs
npm verb addNamed "*" is a valid semver range for qs
npm verb addNameRange registry:http://10.2.117.158/sinopia/qs not in flight; fetching
npm verb request uri http://10.2.117.158/sinopia/qs
npm verb request no auth needed
npm info attempt registry request try #1 at 00:55:52
npm verb request id 1013938c0b7c9c20
npm http request GET http://10.2.117.158/sinopia/qs
npm http 404 http://10.2.117.158/sinopia/qs
npm verb headers { server: 'nginx/1.0.15',
npm verb headers   date: 'Tue, 03 Nov 2015 17:56:20 GMT',
npm verb headers   'content-type': 'application/json; charset=utf-8',
npm verb headers   connection: 'keep-alive',
npm verb headers   'x-powered-by': 'Sinopia/1.4.0',
npm verb headers   'content-length': '39',
npm verb headers   etag: 'W/"27-ae46d555"',
npm verb headers   vary: 'Accept-Encoding',
npm verb headers   'x-status-cat': 'http://flic.kr/p/aV6juR' }
npm verb stack Error: version not found: qs : qs
npm verb stack     at CachingRegistryClient.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:247:14)
npm verb stack     at Request._callback (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:170:14)
npm verb stack     at Request.self.callback (/usr/local/lib/node_modules/npm/node_modules/request/request.js:197:22)
npm verb stack     at Request.emit (events.js:110:17)
npm verb stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/request/request.js:1050:14)
npm verb stack     at Request.emit (events.js:129:20)
npm verb stack     at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/request/request.js:996:12)
npm verb stack     at IncomingMessage.emit (events.js:129:20)
npm verb stack     at _stream_readable.js:908:16
npm verb stack     at process._tickCallback (node.js:355:11)
npm verb statusCode 404
npm verb pkgid qs
npm verb cwd /Users/akshayakumarsharma/Workspace/myapp
npm ERR! Darwin 15.0.0
npm ERR! argv "node" "/usr/local/bin/npm" "install" "qs" "--verbose"
npm ERR! node v0.12.7
npm ERR! npm  v2.13.0
npm ERR! code E404

npm ERR! 404 version not found: qs : qs
npm ERR! 404
npm ERR! 404 'qs' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of 'myapp'
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.
npm verb exit [ 1, true ]

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/akshayakumarsharma/Workspace/myapp/npm-debug.log

Can you please suggest what should I do?

[deleted]

I've encountered the same issue.
Things don't seem to work very well with a proxy. It serves both css/js resources and the packages themselves from the host and port specified in the listen: section of sinopia's config, even if you're proxying to another port with nginx.

It would be great if those resources were specified in a relative way, so we could take advantage of nginx features such as SSL, buffering, and HTTP auth.

[akshayakrsh]

Hi @deleted, let me clarify something about my http proxy setup.

[sinopia server] ==> [my machine] - no proxy exists between the two machines, they are on the same network.
[sinopia server] ==> [internet] - connected via proxy (http_proxy & https_proxy settings added in config.yaml.

1. Web Interface - I only have port 80 opened in the firewall and nginx uses it. Is there no way, I can fix the web interface issue using nginx?
2. NPM registry lookup - Any ideas? Why on request for a npm package, it doesnt even care about doing a lookup.

[trentearl]

I had the some similar issues, maybe youre not serving the assets (which are located in sinopia_dir/lib.

I also had the problem of running nginx with ssl, and node with http. Sinopia thought I was running http and generated links with http. I fixed this by putting this line in config.yaml:

url_prefix: https://npm.mydomain.com

Everything worked great after that.

[deleted]

@trentearl Awesome! That was the setting I was looking for. Fixed both the web UI assets and the package URLs.

@lordakshaya I think adding an url_prefix line to your config may fix your first problem. I'm not sure what to make of your second one.

[akshayakrsh]

@trentearl this is great. I have got the web interface and npmjs registry look up working. Thank you so much.

I still face one issue, some of my NPM modules are coming from github (thru https://) and its failing. I checked that git proxy is already setup. Any idea, what could be going wrong?

[trentearl]

@lordakshaya I wish I could help, but I dont know. We arent using github.

[akshayakrsh]

@trentearl ok no problem. @rlidwka does sinopia support caching npm modules coming in from git?

[RnbWd]

It's an easy fix, but it modifies the sinopia repo

// for nginx proxy 
  if (process.env.VIRTUAL_HOST) {
    app.enable('trust proxy');
  }

There's a pull request pending, and it's been addressed here: #232

I use a fork of sinopia with docker https://hub.docker.com/r/rnbwd/sinopia/ intended to be used in front of a proxy - but I'd recommend forking your own repo or using the fork from the pull request at the moment - this is just a patchwork fix, and I'd like to try and contribute to this repo.. but it's an extremely complex express app and I found it difficult to modify the config.yaml (although I might have got it to work). The trust proxy patch work better? 😕

[joshlevinson]

Sorry for dredging up an old thread, but I'm having trouble with getting the reverse proxy to work at all.

I can get sinopia to serve the css/js files in browser without changing the listen or url_prefix directives, but when npm (client) tries to fetch the .tgz from the sinopia server, it results in a 404 (url format: http://{server.ip.address.here}/{package-name}/-/{package-name}-3.0.0.tgz )

If I use sinopia on the same machine (i.e. installing npm modules on the server, which fetches over localhost), it works - this is only an issue when making an external machine tries to access sinopia packages via the nginx proxy.

I know the tgz requests get sent to sinopia, because it results in the sinopia-style 404 ({ "error": "file not found" }).

Here's my nginx config:

server {
    listen 80 default_server;
    location / {
        proxy_pass http://127.0.0.1:4873;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }

    location ~ /(?<url>-/.*) {
        #this solves the problem of css/js files not loading
        proxy_pass http://127.0.0.1:4873/$url;
    }
}

[RnbWd]

if you're using https I realized I needed to make this adjustment locally

npm config set strict-ssl false

But if the nginx proxy is on a different machine than sinopia - my method won't help. I'm a little confused about your setup - but I'm hosting everything in the cloud - not localhost, and I use the docker nginx-proxy container on the same server as sinopia - and the configuration is just created on the fly, so I don't actually write any nginx code (or I haven't since the repo stabilized).

But how do you know that sinopia is receiving the packages if it's giving a 404?

edit: you can test it here: https://synctea.com

you have to disable strict-ssl locally to use it - I'm just hosting it on a $5 digital ocean - curious if the same error happens. (it's legit https I don't know why npm doesn't accept it - git / curl / browsers work fine)

[RnbWd]

actually yeah my server just pooped trying to download a hyperfs - i don't think sinopia (at least with the current npm) resolves all the paths to repos with more diverse assets

[sullyme]

Any news for nginx and 'package on npmjs.org' is not in the npm registry problem?

I'm using nginx for https connections to my server.

[tjsampson]

I've setup my sinopia service behind nginx proxy with SSL/TLS. I stumbled on this post when trying to resolve it. Not sure if my configuration will help you or not, but here they are.

NGINX conf

server {
        listen                      80;
        server_name         npm.domain.com www.npm.domain.com;
        return          301 https://$server_name$request_uri;
}
server {
        listen                      443 ssl;
        server_name         npm.domain.com www.npm.domain.com;


        ssl_certificate /sinopia/ssl/npm.domain.com.crt;
        ssl_certificate_key /sinopia/ssl/npm.domain.com.key;
        ssl on;
        ssl_session_cache  builtin:1000  shared:SSL:10m;
        ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;

        location / {
        proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;

            proxy_pass          http://localhost:4873;
            proxy_read_timeout  90;
            proxy_redirect      http://localhost:4873 https://npm.domain.com;
    }

}

Sinopia Conf

storage: ./storage

auth:
  htpasswd:
    file: ./htpasswd

uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    # scoped packages
    access: $all
    publish: $authenticated

  '*':
    # allow all users (including non-authenticated users) to read and
    # publish all packages
    #
    # you can specify usernames/groupnames (depending on your auth plugin)
    # and three keywords: "$all", "$anonymous", "$authenticated"
    access: $all

    # allow all known users to publish packages
    # (anyone can register by default, remember?)
    publish: $authenticated

    # if package is not available locally, proxy requests to 'npmjs' registry
    proxy: npmjs

#####################################################################
# Advanced settings
#####################################################################

# if you use nginx with custom path, use this to override links
url_prefix: https://npm.domain.com

listen:
  - 0.0.0.0:4873              # listen on all addresses (INADDR_ANY)

https:
  key: /sinopia/ssl/npm.domain.com.key
  cert: /sinopia/ssl/npm.domain.com.crt

logs:
  - {type: stdout, format: pretty, level: http}

[RnbWd]

Right now, I'm thinking a lot about security questions. Two years ago this
was an obscure project, security wasnt even on my radar a month ago, but
now it seems to dominate my thinking. I'm curious if there are security
risks associated with hosting sinopia behind nginx-proxy. There might be,
so i think security in general should be something we discuss as we
determine whether or not sinopia - as it continues to collect technical
debt - presents any security risks. I'll admit I've watched a few too many
black hat talks in recent weeks, but I never really thought about it. And
I'm wondering if security is on anybody's radar?
On Sun, Sep 11, 2016 at 9:42 PM Troy Sampson notifications@github.com
wrote:

I've setup my sinopia service behind nginx proxy with SSL/TLS. I stumbled
on this post when trying to resolve it. Not sure if my configuration will
help you or not, but here they are.

NGINX conf

server {
listen 80;
server_name npm.domain.com www.npm.domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name npm.domain.com www.npm.domain.com;

    ssl_certificate /sinopia/ssl/npm.domain.com.crt;
    ssl_certificate_key /sinopia/ssl/npm.domain.com.key;
    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    location / {
    proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;

        proxy_pass          http://localhost:4873;
        proxy_read_timeout  90;
        proxy_redirect      http://localhost:4873 https://npm.domain.com;
}

}

Sinopia Conf

storage: ./storage

auth:
htpasswd:
file: ./htpasswd

uplinks:
npmjs:
url: https://registry.npmjs.org/

packages:
'@/':
# scoped packages
access: $all
publish: $authenticated

'*':
# allow all users (including non-authenticated users) to read and
# publish all packages
#
# you can specify usernames/groupnames (depending on your auth plugin)
# and three keywords: "$all", "$anonymous", "$authenticated"
access: $all

# allow all known users to publish packages
# (anyone can register by default, remember?)
publish: $authenticated

# if package is not available locally, proxy requests to 'npmjs' registry
proxy: npmjs

#####################################################################

Advanced settings

#####################################################################

if you use nginx with custom path, use this to override links

url_prefix: https://npm.domain.com

listen:

• 0.0.0.0:4873 # listen on all addresses (INADDR_ANY)

https:
key: /sinopia/ssl/npm.domain.com.key
cert: /sinopia/ssl/npm.domain.com.crt

logs:

• {type: stdout, format: pretty, level: http}

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#335 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFHxwV4cre_zPqnx-g0AI_TgYGT_5R8pks5qpNg_gaJpZM4GbIpe
.