Skip to content

lol #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 95 commits into
base: main
Choose a base branch
from
Open

lol #4

wants to merge 95 commits into from

Conversation

Butanium
Copy link
Collaborator

No description provided.

mntss and others added 30 commits October 14, 2024 20:59
@mntss
Fix BatchTopKSAE training
4aea538
@adamkarvonen
Add a simple end to end test
fe54b00
@adamkarvonen
Rename input to inputs per nnsight 0.3.0
9ed4af2
@adamkarvonen
Complete nnsight 0.2 to 0.3 changes
807f6ef
@adamkarvonen
Fix frac_alive calculation, perform evaluation over multiple batches
dc30720
@adamkarvonen
Obtain better test results using multiple batches
067bf7b
@adamkarvonen
Add early stopping in forward pass
05fe179
@adamkarvonen
Change save_steps to a list of ints
f1b9b80
@adamkarvonen
Check for is_tuple to support mlp / attn submodules
d350415
@adamkarvonen
Merge pull request #26 from mntss/batchtokp_aux_fix
2ec1890 
Fix BatchTopKSAE training
@adamkarvonen
Merge pull request #30 from adamkarvonen/add_tests
c4eed3c 
Add end to end test, upgrade nnsight to support 0.3.0, fix bugs
@adamkarvonen
Ensure activation buffer has the correct dtype
d416eab
@adamkarvonen
Fix JumpReLU training and loading
552a8c2
@adamkarvonen
Begin creation of demo script
712eb98
@adamkarvonen
Modularize demo script
dcc02f0
@adamkarvonen
Track threshold for batchtopk, rename for consistency
32d198f
@adamkarvonen
Track thresholds for topk and batchtopk during training
b5821fd
@adamkarvonen
Remove demo script and graphing notebook
57f451b
@adamkarvonen
Add option to normalize dataset activations
81968f2
@adamkarvonen
Fix topk bfloat16 dtype error
488a154
@adamkarvonen
Add bias scaling to topk saes
484ca01
@adamkarvonen
Use the correct standard SAE reconstruction loss, initialize W_dec to…
8b95ec9 
… W_enc.T
@adamkarvonen
Also scale topk thresholds when scaling biases
efd76b1
@adamkarvonen
Merge pull request #31 from saprmarks/add_demo
67a7857 
Add option to normalize dataset, track thresholds for TopK SAEs, Fix Standard SAE
@adamkarvonen
Remove leftover variable, update expected results with standard SAE i…
9687bb9 
…mprovements
@adamkarvonen
Track lr decay implementation
f0bb66d
@adamkarvonen
Clean up lr decay
e0db40b
@adamkarvonen
Add sparsity warmup for trainers with a sparsity penalty
911b958
@adamkarvonen
Merge pull request #32 from saprmarks/add_sparsity_warmup
a11670f 
Add sparsity warmup
@adamkarvonen
Standardize learning rate and sparsity schedules
a2d6c43
adamkarvonen and others added 30 commits January 13, 2025 20:22
@adamkarvonen
Fix incorrect auxk logging name
784a62a
@adamkarvonen
Fix matryoshka spelling
aa45bf6
@adamkarvonen
Use torch.split() instead of direct indexing for 25% speedup
505a445
@adamkarvonen
simplify matryoshka loss
43421f5
@chanind
feat: pypi packaging and auto-release with semantic release
0ff8888
@saprmarks
Merge pull request #37 from chanind/pypi-package
a711efe 
feat: pypi packaging and auto-release with semantic release
0.1.0
07975f7 
Automatically generated by python-semantic-release
@adamkarvonen
Add a pytorch activation buffer, enable model truncation
944edd1
@adamkarvonen
Add better dataset generators
c644ccd
@adamkarvonen
Add optional backup step
17a41c7
@adamkarvonen
add activault buffer implementation
fe9d8c7
@adamkarvonen
Merge pull request #42 from saprmarks/mixed_datasets
d639166 
Add mixed datasets, model truncation, new activation buffers
@SewoongLee
Update README.md
bdf91b0
@SewoongLee
Update README.md
b560b65
@SewoongLee
Update README.md
02b5343
@andyrdt
assert right padding for remove_bos logic
c7b2527
@andyrdt
mask out bos activations
59abc88
@adamkarvonen
Merge pull request #44 from andyrdt/andyrdt/bos_bugfix
b2bb4b5 
safer remove_bos logic
@Noam-Diamant
added load_dictionary to readme.md in order to load different SAEs
6ca39b5
@canrager
add handling of device and dtype to IdentityDict
61ac634
@adamkarvonen
Add optional k-annealing
ee08651
@adamkarvonen
Add qwen3
c3e4bf4
@adamkarvonen
Handle remove_bos if tokenizer does not contain a bos token
8f87957
@adamkarvonen
Merge pull request #48 from Noam-Diamant/fixes
4d2d46b 
added load_dictionary to readme.md in order to load different SAEs
@adamkarvonen
Merge pull request #43 from SewoongLee/main
eb07533 
Fix broken links in README
@adamkarvonen
Merge pull request #50 from saprmarks/topk_annealing
f4abe32 
Topk annealing
@adamkarvonen
Fix demo
227da89
@adamkarvonen
Add optional mask for high norm tokens
b8bb4ae
@adamkarvonen
Improve variable name
1d2737a
@adamkarvonen
Merge pull request #52 from saprmarks/remove-high-norm
60ec6bf 
Add optional mask for high norm tokens
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@Butanium @mntss @adamkarvonen @chanind @saprmarks @SewoongLee @andyrdt @Noam-Diamant @canrager