Skip to content

Commit 63961b6

Browse files
CannonLockCannonLock
committed
Merge in Master
2 parents c651ae0 + 6fb3663 commit 63961b6

File tree

375 files changed

+13095
-1369
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

375 files changed

+13095
-1369
lines changed

‎.github/workflows/build_deploy.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
ruby-version: 2.6
1515
bundler-cache: true
1616
- run: |
17-
./script/cibuild
17+
./script/cibuild production
1818
name: Build web pages
1919
# SSH key recipe from https://www.webfactory.de/blog/use-ssh-key-for-private-repositories-in-github-actions
2020
- run: |

‎.github/workflows/web-preview.yml

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
2+
name: Deploy preview web area
3+
4+
on:
5+
push:
6+
branches: [preview]
7+
8+
jobs:
9+
deploy-preview:
10+
runs-on: ubuntu-latest
11+
if: startsWith(github.repository, 'CHTC/')
12+
steps:
13+
- uses: actions/checkout@v2
14+
with:
15+
fetch-depth: 0
16+
- uses: ruby/setup-ruby@v1
17+
with:
18+
ruby-version: 2.6
19+
bundler-cache: true
20+
- run: |
21+
sed -i "s|baseurl: ''|baseurl: '/web-preview'|" _config.yml
22+
./script/cibuild
23+
name: Build web pages
24+
# SSH key recipe from https://www.webfactory.de/blog/use-ssh-key-for-private-repositories-in-github-actions
25+
- run: |
26+
mkdir -p ~/.ssh
27+
ssh-keyscan github.com >> ~/.ssh/known_hosts
28+
ssh-agent -a "$SSH_AUTH_SOCK" > /dev/null
29+
ssh-add - <<< "${{ secrets.PREVIEW_DEPLOY_KEY }}"
30+
./script/cideploy
31+
env:
32+
#NO_PUSH: 1
33+
TARGET_REPO: CHTC/web-preview
34+
BRANCH: main
35+
SSH_AUTH_SOCK: /tmp/ssh_agent.sock
36+
if: ${{ github.event_name != 'pull_request' && github.repository_owner == 'CHTC' }}
37+
name: Deploy preview web pages on merge/push

‎.htaccess

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Display 404.html on 404
2+
ErrorDocument 404 /404.html
3+
4+
# Redirect ".shtml" requests
5+
RewriteEngine on
6+
RewriteCond %{REQUEST_FILENAME} !-d
7+
RewriteCond %{REQUEST_URI} ^(.+)\.shtml$
8+
RewriteRule ^ %1 [R=permanent,L]

‎404.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
layout: markdown-page
3+
title: 404 Page Not Found
4+
permalink: /404.html
5+
---
6+
7+
The page you were looking for was not found.
8+
9+
**If current file extension is .shtml, change to .html and retry**
10+
11+
If the issue persists, and you believe this was in error, please contact <mailto:[email protected]>

‎DAGenv.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
highlighter: none
3-
layout: default
3+
layout: markdown-page
44
title: Submitting Jobs Using the ChtcRun Package
55
---
66

@@ -47,7 +47,7 @@ Grid and national <a href="http://www.opensciencegrid.org/">Open Science Grid</a
4747
**For Matlab, Python, and R jobs, it is first essential to have
4848
completed the steps to compile your code dependencies, according to
4949
our [\"Compiling Matlab, Python and R Code\"
50-
page.](http://chtc.cs.wisc.edu/MATLABandR.shtml)**
50+
page.](/MATLABandR)**
5151

5252
Make a directory within `ChtcRun` to house the project\'s input
5353
files and executables. For purposes of this example, assume that the
@@ -114,7 +114,7 @@ Grid and national <a href="http://www.opensciencegrid.org/">Open Science Grid</a
114114
amounts that should be requested for your jobs. For more information
115115
on testing, Flocking, and Glidein, please see the bottom of our
116116
[\"Run Your First HTC Jobs\"
117-
guide](http://chtc.cs.wisc.edu/helloworld.shtml).
117+
guide](/helloworld).
118118

119119
4. With a current working directory of `ChtcRun`, run `mkdag` with a
120120
job-specific set of command line arguments. The output from from

‎HTC-guides.md

+11-11
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,26 @@
11
---
22
highlighter: none
3-
layout: default
3+
layout: markdown-page
44
title: HTC System Use Guides
55
---
66

77
This page lists all of our online resources for running work on the
88
CHTC\'s high throughput system.
99

1010
- Getting Started
11-
- [Running your first CHTC jobs](/helloworld.shtml)
12-
- [Submit file variations](/submit_variations.shtml)
11+
- [Running your first CHTC jobs](helloworld)
12+
- [Submit file variations](submit_variations)
1313
- [Compiling all other code (interactive
14-
jobs)](/inter-submit.shtml)
14+
jobs)](inter-submit)
1515
- How to handle your files and data
16-
- [File Availability options](/file-availability.shtml)
17-
- [File availability with Squid proxy](/file-avail-squid.shtml)
18-
- [File availability with Gluster](/file-avail-gluster.shtml)
16+
- [File Availability options](file-availability)
17+
- [File availability with Squid proxy](file-avail-squid)
18+
- [File availability with Gluster](file-avail-gluster)
1919
- [Using Gluster to stage large
20-
files](/gluster-large-files.shtml)
20+
files](gluster-large-files)
2121
- How to handle your software
2222
- [Submitting Matlab, Python and R jobs using the ChtcRun
23-
environment](/howto_overview.shtml)
24-
- [Submitting Matlab jobs](/matlab-jobs.shtml)
25-
- [Software in Gluster](/gluster-software.shtml)
23+
environment](howto_overview)
24+
- [Submitting Matlab jobs](matlab-jobs)
25+
- [Software in Gluster](gluster-software)
2626
- Running high memory jobs

‎How-Tos.md

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
layout: content
3+
title: How To's
4+
---
5+
6+
<div class="uw-full-row">
7+
<div class="uw-row uw-flex-reverse">
8+
9+
<!-- Body content -->
10+
<div class="uw-col uw-body">
11+
</div>
12+
<aside class="uw-col uw-sidebar">
13+
<button class="uw-button-unstyle uw-side-nav-button"><svg aria-hidden="true" focusable="false"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#uw-symbol-caret-down"></use></svg> Select a section…</button>
14+
15+
<!-- Sidebar nav -->
16+
<div class="uw-side-nav">
17+
<ul>
18+
<li>
19+
<a href="{{ '/get-started' | relative_url }}">Get Started</a>
20+
</li>
21+
<li>
22+
<a href="{{ '/get-help' | relative_url }}">Get Help</a>
23+
</li>
24+
<li>
25+
<a href="{{ '/guides' | relative_url }}">All User Guides</a>
26+
</li>
27+
<li>
28+
<a href="{{ '/use-submit-node' | relative_url }}">Use an HTC Submit Server</a>
29+
</li>
30+
<li>
31+
<a href="{{ '/helloworld' | relative_url }}">Run Your First HTC Jobs</a>
32+
</li>
33+
<li>
34+
<a href="{{ '/howto_overview' | relative_url }}">HTC for MatLab, Python or R</a>
35+
</li>
36+
<li>
37+
<a href="{{ '/hpc-overview' | relative_url }}">Use the HPC Cluster</a>
38+
</li>
39+
<li>
40+
<a href="{{ '/cite-chtc' | relative_url }}">Cite CHTC Resources</a>
41+
</li>
42+
</ul>
43+
</div>
44+
</aside>
45+
</div>
46+
</div>

‎MATLABandR.md

+6-6
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
11
---
22
highlighter: none
3-
layout: default
3+
layout: markdown-page
44
title: Compiling Matlab, Python, and R Code
55
---
66

77

8-
Running MATLAB, Python, or R under CHTC almost always requires a
8+
Running MAgit pTLAB, Python, or R under CHTC almost always requires a
99
two-step process. The first step, described here, compiles or builds the
1010
executable code (R libraries, Python modules, or MATLAB m files)
1111
specific to the job. The second step allows you to submit many jobs
1212
using the built program or program components you\'ve created here. For
1313
this second step, see our guide for setting up and submitting multiple
14-
jobs with the [ChtcRun Package](DAGenv.shtml).
14+
jobs with the [ChtcRun Package](DAGenv).
1515

1616
When and Why to Compile Code for CHTC Jobs
1717
------------------------------------------
@@ -23,7 +23,7 @@ needed. All MATLAB jobs run at the CHTC must be compiled in order to be
2323
run on multiple machines (per Matlab\'s license requirements).\
2424
**All other code compiling for CHTC jobs needs to be completed on one of
2525
our designated \"build machines\" using an interactive submit file and
26-
instructions that you can find [here](inter-submit.shtml).**\
26+
instructions that you can find [here](inter-submit).**\
2727
\
2828
The tools below (chtc\_mcc, chtc\_buildRlibs, and
2929
chtc\_buildPythonmodules) are part of the \"chtc-utils\" suite of tools.
@@ -84,7 +84,7 @@ Our NEW supported Python version (2.7.7) includes the full SciPy Stack
8484
except for iPython, as iPython (interactive) is not compatible with
8585
batch jobs. The following modules and their dependiences are included by
8686
default in our Python 2.7.7 capabilities (when using
87-
[ChtcRun](http://chtc.cs.wisc.edu/DAGenv.shtml)):
87+
[ChtcRun](/DAGenv)):
8888

8989
- numpy-1.8.1
9090
- scipy-0.14.0
@@ -136,7 +136,7 @@ with the command
136136

137137
chtc_buildPythonmodules --help
138138

139-
Setting up your built Python code in the [ChtcRun package](http://chtc.cs.wisc.edu/DAGenv.shtml).
139+
Setting up your built Python code in the [ChtcRun package](/DAGenv).
140140
-------------------------------------------------------------------------------------------------
141141

142142
The ChtcRun folder includes a `Pythonin` directory, which has it\'s own

‎MPIuseguide.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
highlighter: none
3-
layout: default
3+
layout: markdown-page
44
title: MPI Use Guide
55
---
66

@@ -134,5 +134,5 @@ are automatically unloaded.
134134
In order to ensure that your job has the appropriate modules loaded when
135135
it runs, we recommend adding the `module load` command to your submit
136136
file, with the appropriate modules. See our sample submit file in the
137-
[HPC Use Guide](/HPCuseguide.shtml#batch-job) to see what this looks
137+
[HPC Use Guide](HPCuseguide#batch-job) to see what this looks
138138
like.

‎Other-Resources.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
---
2+
layout: content
3+
title: Other Resources
4+
---
5+
6+
<div class="uw-full-row">
7+
<div class="uw-row uw-flex-reverse">
8+
9+
<!-- Body content -->
10+
<div class="uw-col uw-body">
11+
</div>
12+
<aside class="uw-col uw-sidebar">
13+
<button class="uw-button-unstyle uw-side-nav-button"><svg aria-hidden="true" focusable="false"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#uw-symbol-caret-down"></use></svg> Select a section…</button>
14+
15+
<!-- Sidebar nav -->
16+
<div class="uw-side-nav">
17+
<ul>
18+
<li>
19+
<a href="{{ '/gpu-lab' | relative_url }}">CHTC GPU Lab</a>
20+
</li>
21+
<li>
22+
<a href="http://research.cs.wisc.edu/htcondor/">HTCondor Project</a>
23+
</li>
24+
<li>
25+
<a href="http://www.neos-server.org/">NEOS Optimization Service</a>
26+
</li>
27+
<li>
28+
<a href="http://www.opensciencegrid.org/">Open Science Grid</a>
29+
</li>
30+
<li>
31+
<a href="http://wid.wisc.edu/">WID (Wisconsin Institute for Discovery)</a>
32+
</li>
33+
</ul>
34+
</div>
35+
</aside>
36+
</div>
37+
</div>

‎TEMPLATE.shtml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
layout: default
2+
layout: content
33
title: TEMPLATE PAGE
44
---
55

‎_config.yml

+8-2
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
1+
baseurl: ''
2+
exclude: [vendor, deploy_key.enc, TEMPLATE.shtml, archive_html, dev_website]
13

2-
exclude: [vendor, deploy_key.enc, TEMPLATE.shtml]
4+
# Google analytics
5+
google_analytics:
36

47
# Jekyll does not build hidden directories by default.
5-
include: [".well-known"]
8+
include: [".well-known", ".htaccess"]
69

710
markdown_ext : "md,shtml"
811

@@ -15,6 +18,9 @@ defaults:
1518
values:
1619
layout: "default"
1720

21+
sass:
22+
style: compressed
23+
1824
plugins:
1925
- jekyll-redirect-from
2026

‎_data/projects/Amit-Gandhi.yml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
title: Amit Gandhi
2+
image: /images/gandhi.jpg
3+
description: >
4+
I use CHTC to do structural estimation of economic models. These problems boil down to
5+
finding a value in a parameter space that best satisfies some objective criterion that is
6+
derived from economic theory and is a function of the data. The difficulty is that the
7+
criterion is typically a complicated non-linear function of the parameter that requires
8+
some intensive computations for each observation in the data. For problems with lots of
9+
data, this can become unwieldy! Rather than work with an MPI model for distributed computing,
10+
I have found that exploring the landscape of the objective function with the high throughput
11+
CHTC approach allows me to much more quickly gain an understanding of the likely parameter values.
12+
This greatly facilitates the econometric estimation of problems that otherwise would not be feasible.

‎_data/projects/Atlas-Experiment.yml

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
title: Atlas Experiment
2+
image: /images/atlas.jpg
3+
description: <a href="http://atlas.ch"><strong>Atlas Experiment</strong></a>

‎_data/projects/Barry-Van-Veen.yml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
title: Barry Van Veen
2+
image: /images/vanveen.jpg
3+
description: >
4+
The bio-signal processing laboratory develops statistical signal processing
5+
methods for biomedical problems. We use CHTC for casual network modeling of
6+
brain electrical activity. We develop methods for identifying network models
7+
from noninvasive measures of electric/ magnetic fields at the scalp, or
8+
invasive measures of the electric fields at or in the cortex, such as electrocorticography.
9+
Model identification involves high throughput computing applied to large datasets
10+
consisting of hundreds of spatial channels each containing thousands of time samples.

‎_data/projects/CMS-LHS-Compact-Muon-Solenoid.yml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
title: CMS LHC Compact Muon Solenoid
2+
image: /images/LHC.jpg
3+
description: The UW team participating in the Compact Muon Solenoid (CMS)
4+
experiment analyzes petabytes of data from proton-proton collisions in the Large Hadron Collider (LHC). We use
5+
the unprecedented energies of the LHC to study Higgs Boson signatures, Electroweak Physics, and the possibility
6+
of exotic particles beyond the Standard Model of Particle Physics. Important calculations are also performed to
7+
better tune the experiment's trigger system, which is responsible for making nanosecond-scale decisions about
8+
which collisions in the LHC should be recorded for further analysis.

‎_data/projects/David-C-Schwartz.yml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
title: David C. Schwartz
2+
image: /images/schwartz.jpg
3+
description: >
4+
David uses the CHTC to comprehensively analyze human and cancer genomes. CHTC
5+
provices enough throughput for this project to map data representing the
6+
equivalent of one human genome in 90 minutes. See the
7+
<a href="http://www.lmcg.wisc.edu/">LMCG (Laboratory for molecular and Computational Genomics) web site</a>
8+
for project details.

‎_data/projects/Hazy-Research-Group.yml

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
title: Hazy Research Group
2+
image: /images/hazy.jpg
3+
description: >
4+
The Hazy Research Group of the department of Computer Sciences is led by <strong>Christopher M. Re</strong>,
5+
with interests in large-scale, and deep data analytics. A machine reading system is a large software
6+
system that extracts information and knowledge buried in raw data such as text, tables, figures,
7+
and scanned documents. For example, it can extract
8+
facts like "Barack Obama wins the 2012 election" from news articles, or "Barnett Formation contains 6%
9+
Carbon" from geology journal articles. To extract this kind of information, a machine reading system
10+
requires deep understanding and statistical analytics over large document corpora. In the Hazy Research
11+
Group, we are building a machine reading system that supports scientific applications like
12+
<a href="http://hazy.cs.wisc.edu/hazy/geo">GeoDeepDive</a>, and many other projects. Please visit our
13+
<a href="http://www.youtube.com/HazyResearch">YouTube Channel</a> for video overviews of our projects.
14+
We leverage the resources of the CHTC, and the national Open Science Grid to enable our machine reading
15+
system to quickly perform a whole host of computationally expensive tasks like statistical linguistic
16+
processing, speech-to-text transcription, and optical character recognition (OCR). For example, on a crawl of
17+
500 million web pages, we estimated that our deep linguistic parsing would take more than 5 years on a single
18+
machine. With the help of CHTC, we were able to do it in just 1 week! Similarly, on a recent batch of
19+
30,000 geology journal articles, we estimated that the OCR task would take 34 years on a single machine.
20+
With CHTC, it took about 2 weeks. Thanks CHTC!

‎_data/projects/John-L-Markley.yml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
title: John L. Markley
2+
image: /images/markley.jpg
3+
description: >
4+
John uses the CHTC for research in connection with the
5+
<a href="http://www.bmrb.wisc.edu">Biological Magnetic Resonance Data Bank</a> (BMRB).

‎_data/projects/Juan-De-Pablo.yml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
title: Juan De Pablo
2+
image: /images/depablo.jpg
3+
description: Juan de Pablo and the <A HREF="http://www.engr.wisc.edu/groups/mtsm/">Molecular Thermodynamics and Statistical
4+
Mechanics Research Group</A> use computational resources provided by the CHTC to predict the motions of
5+
macroscopic objects through simulations of what their microscopic particles are doing.

0 commit comments

Comments
 (0)